Privacy Artificial Intelligence

View on GitHub

Random Seed

In this notebook, we provide a simple example of how to perform an experiment in a federated environment, with the help of this framework. We are going to use a popular dataset to start the experimentation in a federated environment. The framework provides some functions for loading the Emnist Digits dataset.

This notebook is a copy of the A Simple Experiment notebook. The difference is that, here, we set a seed using Reproducibility Singleton Class, in order to ensure the reproducibility of the experiment. If you execute this experiment many times, you should obtain the same results.

from shfl.private.reproducibility import Reproducibility

# Server
Reproducibility(1234)

# In case of client
# Reproducibility.get_instance().set_seed(ID)
<shfl.private.reproducibility.Reproducibility at 0x10f52fa90>
import matplotlib.pyplot as plt
import shfl

database = shfl.data_base.Emnist()
train_data, train_labels, test_data, test_labels = database.load_data()

Let's inspect some properties of the loaded data.

print(len(train_data))
print(len(test_data))
print(type(train_data[0]))
train_data[0].shape
240000
40000
<class 'numpy.ndarray'>

(28, 28)

So, as we have seen, our dataset is composed of a set of matrices that are 28 by 28. Before starting with the federated scenario, we can take a look at a sample in the training data.

plt.imshow(train_data[0])

png

We are going to simulate a federated learning scenario with a set of client nodes containing private data, and a central server that will be responsible for coordinating the different clients. But, first of all, we have to simulate the data contained in every client. In order to do that, we are going to use the previously loaded dataset. The assumption in this example is that the data is distributed as a set of independent and identically distributed random variables, with every node having approximately the same amount of data. There are a set of different possibilities for distributing the data. The distribution of the data is one of the factors that can have most impact on a federated algorithm. Therefore, the framework has some of the most common distributions implemented, which allows you easily to experiment with different situations. In Sampling Methods, you can dig into the options that the framework provides, at the moment.

iid_distribution = shfl.data_distribution.IidDataDistribution(database)
federated_data, test_data, test_labels = iid_distribution.get_federated_data(num_nodes=20, percent=10)

That's it! We have created federated data from the Emnist dataset using 20 nodes and 10 percent of the available data. This data is distributed to a set of data nodes in the form of private data. Let's learn a little more about the federated data.

print(type(federated_data))
print(federated_data.num_nodes())
federated_data[0].private_data
<class 'shfl.private.federated_operation.FederatedData'>
20
Node private data, you can see the data for debug purposes but the data remains in the node
<class 'dict'>
{'4552063248': <shfl.private.data.LabeledData object at 0x150676590>}

As we can see, private data in a node is not directly accessible, but the framework provides mechanisms to use this data in a machine learning model. A federated learning algorithm is defined by a machine learning model, locally deployed in each node, that learns from the respective node’s private data and an aggregating mechanism to aggregate the different model parameters uploaded by the client nodes to a central node. In this example, we will use a deep learning model using Keras to build it. The framework provides classes on using TensorFlow (see TensorFlow Model) and Keras models in a federated learning scenario, your only job is to create a function acting as model builder. Moreover, the framework provides classes on using pretrained TensorFlow and Keras models (see Pretrained Models). In this example build a Keras learning model.

import tensorflow as tf

def model_builder():
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu', strides=1, input_shape=(28, 28, 1)))
    model.add(tf.keras.layers.MaxPooling2D(pool_size=2, strides=2, padding='valid'))
    model.add(tf.keras.layers.Dropout(0.4))
    model.add(tf.keras.layers.Conv2D(32, kernel_size=(3, 3), padding='same', activation='relu', strides=1))
    model.add(tf.keras.layers.MaxPooling2D(pool_size=2, strides=2, padding='valid'))
    model.add(tf.keras.layers.Dropout(0.3))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Dense(128, activation='relu'))
    model.add(tf.keras.layers.Dropout(0.1))
    model.add(tf.keras.layers.Dense(64, activation='relu'))
    model.add(tf.keras.layers.Dense(10, activation='softmax'))

    model.compile(optimizer="rmsprop", loss="categorical_crossentropy", metrics=["accuracy"])

    return shfl.model.DeepLearningModel(model)

Now, the only piece missing is the aggregation operator. Nevertheless, the framework provides some aggregation operators that we can use. In the following piece of code, we define the federated aggregation mechanism. Moreover, we define the federated government based on the Keras learning model, the federated data and the aggregation mechanism.

aggregator = shfl.federated_aggregator.FedAvgAggregator()
federated_government = shfl.federated_government.FederatedGovernment(model_builder, federated_data, aggregator)

If you want to see all the aggregation operators, you can check out the Aggregation Operators notebook. Before running the algorithm, we want to apply a transformation to the data. A good practise is to define a federated operation that will ensure that the transformation is applied to the federated data in all the client nodes. We want to reshape the data, so we define the following FederatedTransformation.

import numpy as np

class Reshape(shfl.private.FederatedTransformation):

    def apply(self, labeled_data):
        labeled_data.data = np.reshape(labeled_data.data, (labeled_data.data.shape[0], labeled_data.data.shape[1], labeled_data.data.shape[2],1))

shfl.private.federated_operation.apply_federated_transformation(federated_data, Reshape())

In addition, we want to normalize the data. We define a federated transformation using mean and standard deviation (std) parameters. We use the mean and std estimated from the training set in this example. Although the ideal parameters would be an aggregation of the mean and std of each client's training datasets, we use the mean and std of the global dataset as a simple approximation.

import numpy as np

class Normalize(shfl.private.FederatedTransformation):

    def __init__(self, mean, std):
        self.__mean = mean
        self.__std = std

    def apply(self, labeled_data):
        labeled_data.data = (labeled_data.data - self.__mean)/self.__std


mean = np.mean(train_data.data)
std = np.std(train_data.data)
shfl.private.federated_operation.apply_federated_transformation(federated_data, Normalize(mean, std))

We are now ready to execute our federated learning algorithm.

test_data = np.reshape(test_data, (test_data.shape[0], test_data.shape[1], test_data.shape[2],1))
federated_government.run_rounds(3, test_data, test_labels)
Accuracy round 0
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c667250>: [26.00153923034668, 0.7081500291824341]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150676b50>: [16.699405670166016, 0.7995499968528748]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150676990>: [17.58183479309082, 0.7689999938011169]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150676f90>: [28.322412490844727, 0.6869750022888184]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15062e790>: [23.34100914001465, 0.7476500272750854]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c294d10>: [27.00540542602539, 0.7288249731063843]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c2944d0>: [24.097013473510742, 0.7230499982833862]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c294a90>: [17.562694549560547, 0.7771250009536743]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cdcd0>: [19.65399932861328, 0.7558500170707703]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cd150>: [28.578384399414062, 0.7000749707221985]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cd050>: [27.31490707397461, 0.7062000036239624]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cd750>: [41.24004364013672, 0.6171500086784363]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613e10>: [31.63262176513672, 0.666450023651123]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613690>: [21.67351722717285, 0.7437499761581421]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613890>: [28.882843017578125, 0.7445250153541565]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613cd0>: [25.49474334716797, 0.7039999961853027]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613710>: [61.66909408569336, 0.5647249817848206]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150629810>: [25.390243530273438, 0.7035499811172485]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150629690>: [87.99385070800781, 0.4817500114440918]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15066add0>: [37.62916564941406, 0.6604250073432922]
Global model test performance : [16.592201232910156, 0.7207499742507935]



Accuracy round 1
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c667250>: [14.67207145690918, 0.8634499907493591]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150676b50>: [14.871051788330078, 0.8428500294685364]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150676990>: [17.46796226501465, 0.8440750241279602]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150676f90>: [21.917238235473633, 0.8354499936103821]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15062e790>: [16.510255813598633, 0.8461250066757202]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c294d10>: [21.896974563598633, 0.8167499899864197]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c2944d0>: [19.574256896972656, 0.8292499780654907]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c294a90>: [19.2028865814209, 0.8241750001907349]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cdcd0>: [15.080036163330078, 0.8674499988555908]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cd150>: [19.943450927734375, 0.8134499788284302]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cd050>: [21.701570510864258, 0.8273249864578247]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cd750>: [21.288379669189453, 0.8190000057220459]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613e10>: [18.433635711669922, 0.8406749963760376]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613690>: [33.29237365722656, 0.7657750248908997]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613890>: [19.49696159362793, 0.8552250266075134]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613cd0>: [21.063447952270508, 0.8140749931335449]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613710>: [26.86553192138672, 0.7778750061988831]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150629810>: [19.385807037353516, 0.8181750178337097]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150629690>: [19.066486358642578, 0.8414000272750854]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15066add0>: [17.992136001586914, 0.8444250226020813]
Global model test performance : [12.931232452392578, 0.8767750263214111]



Accuracy round 2
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c667250>: [17.962289810180664, 0.8699250221252441]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150676b50>: [16.878210067749023, 0.871399998664856]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150676990>: [20.96730613708496, 0.8471750020980835]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150676f90>: [30.877519607543945, 0.8162000179290771]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15062e790>: [16.430892944335938, 0.8608499765396118]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c294d10>: [19.220930099487305, 0.8540499806404114]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c2944d0>: [13.712364196777344, 0.8908249735832214]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14c294a90>: [14.427949905395508, 0.8773750066757202]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cdcd0>: [16.550312042236328, 0.8812249898910522]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cd150>: [25.18115997314453, 0.8327999711036682]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cd050>: [22.31023597717285, 0.8464249968528748]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10e7cd750>: [19.233205795288086, 0.8623499870300293]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613e10>: [15.390209197998047, 0.8830000162124634]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613690>: [17.95242691040039, 0.8794500231742859]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613890>: [13.274240493774414, 0.8999999761581421]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613cd0>: [14.092557907104492, 0.8842250108718872]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150613710>: [16.02014923095703, 0.8596500158309937]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150629810>: [21.988723754882812, 0.8403000235557556]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x150629690>: [18.26984977722168, 0.8748499751091003]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15066add0>: [24.2637882232666, 0.8356000185012817]
Global model test performance : [11.947325706481934, 0.9054250121116638]