# A Simple Experiment

In this notebook, we provide a simple example of how to perform an experiment in a federated environment, with the help of this framework. We are going to use a popular dataset to start the experimentation in a federated environment. The framework provides some functions to load the Emnist Digits dataset.

import matplotlib.pyplot as plt
import shfl

database = shfl.data_base.Emnist()
train_data, train_labels, test_data, test_labels = database.load_data()

Let's inspect some properties of the loaded data.

print(len(train_data))
print(len(test_data))
print(type(train_data[0]))
train_data[0].shape
240000
40000
<class 'numpy.ndarray'>

(28, 28)

So, as we have seen, our dataset is composed of a set of matrices that are 28 by 28. Before starting with the federated scenario, we can take a look at a sample of the training data.

plt.imshow(train_data[0])
<matplotlib.image.AxesImage at 0x10fab5d90>

We are going to simulate a federated learning scenario with a set of client nodes containing private data, and a central server that will be responsible for coordinating the different clients. But, first of all, we have to simulate the data contained in every client. In order to do that, we are going to use the previously loaded dataset . The assumption in this example is that the data is distributed as a set of independent and identically distributed random variables, with every node having approximately the same amount of data. There are a set of different possibilities for distributing the data. The distribution of the data is one of the factors that can have the most impact on a federated algorithm. Therefore, the framework has some of the most common distributions implemented, which allows you to easily experiment with different situations. In Sampling Methods, you can dig into the options that the framework provides, at the moment.

iid_distribution = shfl.data_distribution.IidDataDistribution(database)
federated_data, test_data, test_labels = iid_distribution.get_federated_data(num_nodes=20, percent=10)

That's it! We have created federated data from the Emnist dataset using 20 nodes and 10 percent of the available data. This data is a set of data nodes containing private data.
Let's learn a little more about the federated data.

print(type(federated_data))
print(federated_data.num_nodes())
federated_data[0].private_data
<class 'shfl.private.federated_operation.FederatedData'>
20
Node private data, you can see the data for debug purposes but the data remains in the node
<class 'dict'>
{'4570607568': <shfl.private.data.LabeledData object at 0x15ba57210>}

As we can see, private data in a node is not directly accessible, but the framework provides mechanisms to use this data in a machine learning model. A federated learning algorithm is defined by a machine learning model, locally deployed in each node, that learns from the respective node’s private data and an aggregating mechanism to aggregate the different model parameters uploaded by the client nodes to a central node. In this example, we will use a deep learning model using Keras to build it. The framework provides classes on using TensorFlow (see TensorFlow Model) and Keras models in a federated learning scenario, your only job is to create a function acting as model builder. Moreover, the framework provides classes on using pretrained TensorFlow and Keras models (see Pretrained Model). In this example, we will build a Keras learning model.

import tensorflow as tf

def model_builder():
model = tf.keras.models.Sequential()

model.compile(optimizer="rmsprop", loss="categorical_crossentropy", metrics=["accuracy"])

return shfl.model.DeepLearningModel(model)

Now, the only missing piece is the aggregation operator. Nevertheless, the framework provides some aggregation operators that we can use. In the following piece of code, we define the federated aggregation mechanism. Moreover, we define the federated government based on the Keras learning model, the federated data, and the aggregation mechanism.

aggregator = shfl.federated_aggregator.FedAvgAggregator()
federated_government = shfl.federated_government.FederatedGovernment(model_builder, federated_data, aggregator)

If you want to see all the aggregation operators, you can check out the Aggregation Operators notebook. Before running the algorithm, we want to apply a transformation to the data. A good practice is to define a federated operation that will ensure that the transformation is applied to the federated data in all the client nodes. We want to reshape the data, so we define the following FederatedTransformation.

import numpy as np

class Reshape(shfl.private.FederatedTransformation):

def apply(self, labeled_data):
labeled_data.data = np.reshape(labeled_data.data, (labeled_data.data.shape[0], labeled_data.data.shape[1], labeled_data.data.shape[2],1))

shfl.private.federated_operation.apply_federated_transformation(federated_data, Reshape())

In addition, we want to normalize the data. We define a federated transformation using mean and standard deviation (std) parameters. We use the mean and std estimated from the training set, in this example. Although the ideal parameters would be an aggregation of the mean and std of each client's training datasets, we use the mean and std of the global dataset as a simple approximation.

import numpy as np

class Normalize(shfl.private.FederatedTransformation):

def __init__(self, mean, std):
self.__mean = mean
self.__std = std

def apply(self, labeled_data):
labeled_data.data = (labeled_data.data - self.__mean)/self.__std

mean = np.mean(train_data.data)
std = np.std(train_data.data)
shfl.private.federated_operation.apply_federated_transformation(federated_data, Normalize(mean, std))

We are now ready to execute our federated learning algorithm.

test_data = np.reshape(test_data, (test_data.shape[0], test_data.shape[1], test_data.shape[2],1))
federated_government.run_rounds(3, test_data, test_labels)
Accuracy round 0
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14d8ad450>: [18.09648323059082, 0.7206249833106995]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba57590>: [32.75648498535156, 0.6452249884605408]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba574d0>: [24.201391220092773, 0.7250499725341797]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b050>: [13.68442153930664, 0.8115249872207642]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4be10>: [26.784080505371094, 0.7102749943733215]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b650>: [34.55759811401367, 0.6540499925613403]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b150>: [39.525760650634766, 0.5778999924659729]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4bc90>: [19.434309005737305, 0.7458249926567078]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b710>: [19.789508819580078, 0.7518249750137329]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfdd0>: [30.314546585083008, 0.6472749710083008]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf2d0>: [20.770736694335938, 0.7268000245094299]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf810>: [38.365577697753906, 0.6349499821662903]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfa50>: [36.99231719970703, 0.6905750036239624]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf750>: [21.864755630493164, 0.7243750095367432]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfcd0>: [19.3663272857666, 0.75]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf450>: [17.699010848999023, 0.7736499905586243]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf9d0>: [23.664466857910156, 0.7380499839782715]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf290>: [27.369855880737305, 0.7390499711036682]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfd10>: [25.219135284423828, 0.6975499987602234]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfc90>: [61.23874282836914, 0.4614500105381012]
Global model test performance : [14.39564037322998, 0.6877999901771545]

Accuracy round 1
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14d8ad450>: [20.06230926513672, 0.818274974822998]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba57590>: [23.574905395507812, 0.7925249934196472]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba574d0>: [28.298696517944336, 0.8002499938011169]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b050>: [21.95953369140625, 0.8229249715805054]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4be10>: [29.557918548583984, 0.753350019454956]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b650>: [19.7932186126709, 0.831250011920929]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b150>: [27.292728424072266, 0.7688249945640564]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4bc90>: [30.920621871948242, 0.7785999774932861]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b710>: [34.21841812133789, 0.7615000009536743]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfdd0>: [22.511592864990234, 0.8074749708175659]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf2d0>: [29.61403465270996, 0.736050009727478]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf810>: [47.81975173950195, 0.7179499864578247]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfa50>: [16.936946868896484, 0.8608999848365784]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf750>: [20.120389938354492, 0.8098499774932861]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfcd0>: [18.86663818359375, 0.8364999890327454]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf450>: [15.217225074768066, 0.8532999753952026]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf9d0>: [25.674537658691406, 0.8096749782562256]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf290>: [19.799713134765625, 0.8374249935150146]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfd10>: [25.563030242919922, 0.7856249809265137]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfc90>: [39.45299530029297, 0.7161250114440918]
Global model test performance : [16.08136558532715, 0.8450000286102295]

Accuracy round 2
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x14d8ad450>: [22.781232833862305, 0.8046000003814697]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba57590>: [22.383586883544922, 0.8346499800682068]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba574d0>: [21.832551956176758, 0.8344500064849854]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b050>: [16.262527465820312, 0.8795750141143799]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4be10>: [28.056962966918945, 0.7852749824523926]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b650>: [18.04791831970215, 0.8575500249862671]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b150>: [29.395015716552734, 0.7898499965667725]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4bc90>: [16.509410858154297, 0.8737999796867371]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15ba4b710>: [17.030681610107422, 0.856024980545044]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfdd0>: [24.882532119750977, 0.8007000088691711]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf2d0>: [25.37504768371582, 0.8094499707221985]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf810>: [22.512752532958984, 0.8481749892234802]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfa50>: [16.046655654907227, 0.8827750086784363]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf750>: [23.943363189697266, 0.8252999782562256]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfcd0>: [17.773818969726562, 0.8460999727249146]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf450>: [21.868436813354492, 0.8446499705314636]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf9d0>: [34.3828239440918, 0.7700999975204468]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babf290>: [19.373605728149414, 0.8647000193595886]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfd10>: [23.9183292388916, 0.845674991607666]
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x15babfc90>: [40.32728576660156, 0.725724995136261]
Global model test performance : [17.473241806030273, 0.857574999332428]