Inteligencia Artificial de Privacidad

Ver en GitHub

Federated Learning: Classification by Logistic Regression Model

Here, we explain how to set up a federated classification experiment using a Logistic Regression model. Results from the federated learning are compared to the (non-federated) centralized learning. Moreover, we also show how the addition of differential privacy affects the performance of the federated model. In these examples, we will generate synthetic data for the classification task. In particular, we will start from a two-dimensional case, since with only two features, we are able to easily plot the samples and the decision boundaries computed by the classifier. After that, we will repeat the experiment by adding more features and classes to the synthetic database.

Two-feature case:

We generate the data using the make_classification function from scikit-learn:

import shfl
from shfl.data_base.data_base import LabeledDatabase
from sklearn.datasets import make_classification
import numpy as np
from shfl.private.reproducibility import Reproducibility

# Comment to turn off reproducibility:
Reproducibility(1234)

# Create database:
n_features = 2
n_classes = 3
n_samples = 500
data, labels = make_classification(
    n_samples=n_samples, n_features=n_features, n_informative=2,
    n_redundant=0, n_repeated=0, n_classes=n_classes,
    n_clusters_per_class=1, weights=None, flip_y=0.1, class_sep=0.4, random_state=1234)
database = LabeledDatabase(data, labels)

train_data, train_labels, test_data, test_labels = database.load_data()

print("Shape of training and test data: " + str(train_data.shape) + str(test_data.shape))
print("Shape of training and test labels: " + str(train_labels.shape) + str(test_labels.shape))
print(train_data[0,:])
Shape of training and test data: (400, 2)(100, 2)
Shape of training and test labels: (400,)(100,)
[-1.12355796 -1.1096599 ]

As previously mentioned, in this two-feature case, it is beneficial to visualize the results. For that purpose, we will use the following function:

import matplotlib.pyplot as plt

def plot_2D_decision_boundary(model, data, labels, title=None):
    # Step size of the mesh. The smaller it is, better the quality
    h = .02
    # Color map
    cmap = plt.cm.Set1

    # Plot the decision boundary. For that, we will assign a color to each
    x_min, x_max = data[:, 0].min() - 1, data[:, 0].max() + 1
    y_min, y_max = data[:, 1].min() - 1, data[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    # Obtain labels for each point in mesh. Use last trained model.
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    fig, ax = plt.subplots(figsize=(9,6))
    plt.clf()
    plt.imshow(Z, interpolation='nearest',
               extent=(xx.min(), xx.max(), yy.min(), yy.max()),
               cmap=cmap,
               alpha=0.6,
               aspect='auto', origin='lower')
    # Plot data:
    plt.scatter(data[:, 0], data[:, 1], c=labels, cmap=cmap, s=40, marker='o')

    plt.title(title, fontsize=18)
    plt.xlim(x_min, x_max)
    plt.ylim(y_min, y_max)
    plt.xlabel('Feature 1', fontsize=18)
    plt.ylabel('Feature 2', fontsize=18)
    plt.tick_params(labelsize=15)

Running the Federated model
After defining the data, we are ready to run our model in a federated configuration. We distribute the data over the nodes, assuming the data is IID. Next, we define the aggregation of the federated outputs to be the average of the client models.

The Sherpa.ai Federated Learning and Differential Privacy Framework offers support for the Logistic Regression model from scikit-learn. The user must specify, in advance, the number of features and the target classes: the assumption for this Federated Logistic Regression example is that each client's data possesses at least one sample of each class. Otherwise, each node might train a different classification problem, and it would be problematic to aggregate the global model. Setting a model's state parameter to warm_start:True tells the clients to restart the training from the federated round update. To assess the performance, we compute the Balanced Accuracy and the Kohen Kappa scores (see metrics):

from shfl.model.logistic_regression_model import LogisticRegressionModel

# Distribute data over the nodes:
iid_distribution = shfl.data_distribution.IidDataDistribution(database)
n_clients = 5
federated_data, test_data, test_labels = iid_distribution.get_federated_data(num_nodes=n_clients, percent=100)
aggregator = shfl.federated_aggregator.FedAvgAggregator()

# Define the model:
classes = np.unique(train_labels)
def model_builder():
    model = LogisticRegressionModel(n_features=n_features, classes=classes, model_inputs={'warm_start':True})
    return model

# Run the federated experiment:
federated_government = shfl.federated_government.FederatedGovernment(model_builder, federated_data, aggregator)
federated_government.run_rounds(n=2, test_data=test_data, test_label=test_labels)
Accuracy round 0
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c050>: (0.580203823953824, 0.3705035971223022)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144064150>: (0.5940386002886003, 0.3879683534855948)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c1d0>: (0.5625270562770562, 0.34308748880262774)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c2d0>: (0.5486381673881674, 0.32442576189761296)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c450>: (0.4750901875901876, 0.21639541892706438)
Global model test performance : (0.6117694805194805, 0.4159053467125956)



Accuracy round 1
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c050>: (0.580203823953824, 0.3705035971223022)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144064150>: (0.5940386002886003, 0.3879683534855948)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c1d0>: (0.5625270562770562, 0.34308748880262774)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c2d0>: (0.5486381673881674, 0.32442576189761296)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c450>: (0.4750901875901876, 0.21639541892706438)
Global model test performance : (0.6117694805194805, 0.4159053467125956)

It can be observed that the performance of the federated global model is generally superior, with respect to the performance of each node, thus, the federated learning approach proves to be beneficial. Moreover, since no or little performance difference is observed between the federated rounds, we can conclude that the classification problem converges very early, in this setting, and no further rounds are required. This might be due to the IID nature of client data when performing classification: each node gets a representative chunk of data and thus the local models are similar.

Comparison to centralized training
The performance of federated global model is comparable to the performance of the model trained on centralized data, and it produces similar decision boundaries:

# Train model on centralized data for comparison:
model_centralized = LogisticRegressionModel(n_features=n_features, classes=classes)
model_centralized.train(train_data, train_labels)
print('Centralized test performance: ' + str(model_centralized.evaluate(test_data, test_labels)))

# Plot decision boundaries and test data for Federated and (non-Federated) centralized cases:
if n_features == 2:
    plot_2D_decision_boundary(federated_government.global_model, test_data, test_labels, title = "Federated Logistic regression. Test data are shown.")
    plot_2D_decision_boundary(model_centralized._model, test_data, test_labels, title = "Centralized Logistic regression. Test data are shown.")    
Centralized test performance: (0.6221861471861472, 0.43113772455089816)

png

png

Adding Differential Privacy to the Federated model
We want to assess the impact of differential privacy (see The Algorithmic Foundations of Differential Privacy, Section 3.3) on the federated model's performance. In particular, we will use the Laplace mechanism (see also the corresponding Laplace mechanism notebook). The noise added has to be of the same order as the sensitivity of the model's output, i.e., the model parameters of our logistic regression. In general, the sensitivity of a Machine Learning model is difficult to compute (for the Logistic Regression case, refer to Privacy-preserving logistic regression). An alternative strategy may be to estimate the sensitivity through a sampling procedure (e.g. see Rubinstein 2017 and how to use the tools provided by the Sherpa.ai Federated Learning and Differential Privacy Framework in the Linear Regression Notebook). However, be advised that this would guarantee the weaker property of random differential privacy. This approach is convenient, since it allows for the sensitivity estimation of an arbitrary model or a black-box computer function. The Sherpa.ai Federated Learning and Differential Privacy Framework provides this functionality in the class SensitivitySampler.

We need to specify a distribution of the data to sample from. Generally, this requires previous knowledge and/or model assumptions. In order not to make any specific assumptions about the distribution of the dataset, we can choose a uniform distribution. We define our class of ProbabilityDistribution that uniformly samples over a data-frame. We could sample using the training data. However, since in a real case, the training data would be the actual client data, we wouldn't have access to it. Thus, we generate another synthetic dataset for sampling (in a real case, this could be a public database we are able to access):

class UniformDistribution(shfl.differential_privacy.ProbabilityDistribution):
    """
    Implement Uniform sampling over the data
    """
    def __init__(self, sample_data):
        self._sample_data = sample_data

    def sample(self, sample_size):
        row_indices = np.random.randint(low=0, high=self._sample_data.shape[0], size=sample_size, dtype='l')

        return self._sample_data[row_indices, :]

# Create sampling database:
n_samples = 150
sampling_data, sampling_labels = make_classification(
    n_samples=n_samples, n_features=n_features, n_informative=2,
    n_redundant=0, n_repeated=0, n_classes=n_classes,
    n_clusters_per_class=1, weights=None, flip_y=0.1, class_sep=0.1)    
sample_data = np.hstack((sampling_data, sampling_labels.reshape(-1,1)))

The class SensitivitySampler implements the sampling, given a query (i.e., the learning model itself, in this case). We only need to add the get method to our model since it is required by the class SensitivitySampler: it simply trains the model on the input data and outputs the trained parameters. We choose the sensitivity norm to be the L1L_1 norm and we apply the sampling. The value of the sensitivity depends on the number of samples n: the more samples we perform, the more accurate the sensitivity. Indeed, by increasing the number of samples n, the sensitivity becomes more accurate and typically decreases. Note that the sampling could be quite costly, since the query (i.e. the model, in this case) is called, each time.

from shfl.differential_privacy import SensitivitySampler
from shfl.differential_privacy import L1SensitivityNorm

class LogisticRegressionSample(LogisticRegressionModel):

    def get(self, data_array):
        data = data_array[:, 0:-1]
        labels = data_array[:, -1]
        train_model = self.train(data, labels)

        return self.get_model_params()

distribution = UniformDistribution(sample_data)
sampler = SensitivitySampler()

n_samples = 200
max_sensitivity, mean_sensitivity = sampler.sample_sensitivity(
    LogisticRegressionSample(n_features=n_features, classes=classes),
    L1SensitivityNorm(), distribution, n=n_samples, gamma=0.05)
print("Max sensitivity from sampling: " + str(max_sensitivity))
print("Mean sensitivity from sampling: " + str(mean_sensitivity))
Max sensitivity from sampling: 0.5093568568937039
Mean sensitivity from sampling: 0.10604924716420795

Once the model's estimated sensitivity has been obtained, we fix the ϵ\epsilon privacy budget and we can run the privacy-preserving Federated experiment:

from shfl.differential_privacy import LaplaceMechanism

params_access_definition = LaplaceMechanism(sensitivity=max_sensitivity, epsilon=0.5)
federated_governmentDP = shfl.federated_government.FederatedGovernment(
    model_builder, federated_data, aggregator, model_params_access=params_access_definition)

federated_governmentDP.run_rounds(n=2, test_data=test_data, test_label=test_labels)
Accuracy round 0
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c050>: (0.580203823953824, 0.3705035971223022)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144064150>: (0.5940386002886003, 0.3879683534855948)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c1d0>: (0.5625270562770562, 0.34308748880262774)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c2d0>: (0.5486381673881674, 0.32442576189761296)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c450>: (0.4750901875901876, 0.21639541892706438)
Global model test performance : (0.4761002886002887, 0.2163954189270645)



Accuracy round 1
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c050>: (0.580203823953824, 0.3705035971223022)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144064150>: (0.5940386002886003, 0.3879683534855948)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c1d0>: (0.5625270562770562, 0.34308748880262774)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c2d0>: (0.5486381673881674, 0.32442576189761296)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x144b7c450>: (0.4750901875901876, 0.21639541892706438)
Global model test performance : (0.5963654401154401, 0.3970455230630088)

As you might expect, the addition of random noise slightly alters the solution, but the model is still comparable to the non-private federated case:

# Plot decision boundaries and test data for Privacy-preserving Federated case:
if n_features == 2:
    plot_2D_decision_boundary(federated_governmentDP.global_model, test_data, test_labels,
                              title = "Privacy-preserving Federated Logistic regression. Test data are shown.")

png

Note 1: In this case, we only run a few federated rounds. However, in general, you should make sure not to exceed a fixed privacy budget (see how to achieve that in Sherpa.ai Federated Learning and Differential Privacy Framework in the Composition concepts notebook).
Note 2: It is worth mentioning that the above results cannot be considered general. Some factors that considerably influence the classification problem are, for example, the training dataset, the model type, and the differential privacy mechanism used. In fact, the classification problem itself depends on the training data (number of features, whether the classes are separable or not etc.). We strongly encourage users to play with the values for generating the database (such as n_features, n_classes, n_samples, class_sep; see their meanings here), or try different classification datasets, since convergence and accuracy of local and global models can be strongly affected. Moreover, even without changing the dataset, by running the present experiment multiple times (you need to comment the Reproducibility command line code), it is observed that the federated global model may also exhibit slightly better performance when compared to the centralized model (we use random_state input, in order to always produce the same dataset). Similarly, the privacy-preserving federated model might exhibit even better performance, compared to the non-private version. This depends on a) the performance metrics chosen and b) the idiosyncrasy of the specific classification considered here: a small modification to the model's coefficients may alter the class prediction for a few samples.

Case with More Features and Classes:

Below we present a more complex case, introducing more features and classes. When using more than two features, the figures are not plotted. Since the structure of the example is identical to the above, the comments are not repeated:

# Create database:
n_features = 11
n_classes = 5
n_samples = 500
data, labels = make_classification(
    n_samples=n_samples, n_features=n_features, n_informative=4,
    n_redundant=0, n_repeated=0, n_classes=n_classes,
    n_clusters_per_class=2, weights=None, flip_y=0.1, class_sep=0.1, random_state=123)
database = LabeledDatabase(data, labels)

train_data, train_labels, test_data, test_labels = database.load_data()
print("Shape of training and test data: " + str(train_data.shape) + str(test_data.shape))
print("Shape of training and test labels: " + str(train_labels.shape) + str(test_labels.shape))
print(train_data[0,:])
Shape of training and test data: (400, 11)(100, 11)
Shape of training and test labels: (400,)(100,)
[-0.40279912 -0.55942522  0.20294497  0.31181866  0.23188266 -0.1082948
  2.80138858  1.37655706 -0.90024143 -0.52589682 -0.15840162]

Running the Model in a Federated Configuration.

iid_distribution = shfl.data_distribution.IidDataDistribution(database)
federated_data, test_data, test_labels = iid_distribution.get_federated_data(num_nodes=5, percent=100)

classes = np.unique(train_labels)
def model_builder():
    model = LogisticRegressionModel(n_features=n_features, classes=classes, model_inputs={'warm_start':True})
    return model

aggregator = shfl.federated_aggregator.FedAvgAggregator()

federated_government = shfl.federated_government.FederatedGovernment(model_builder, federated_data, aggregator)
federated_government.run_rounds(n=2, test_data=test_data, test_label=test_labels)
Accuracy round 0
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x105872d90>: (0.17103896103896105, -0.03527607361963203)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x105872ed0>: (0.20857586857586857, 0.011758819114335739)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a990>: (0.19086136086136085, -0.008466135458167434)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a810>: (0.23502386502386502, 0.05189620758483027)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a550>: (0.1916072816072816, -0.014349332013854577)
Global model test performance : (0.1401098901098901, -0.05985037406483795)



Accuracy round 1
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x105872d90>: (0.17103896103896105, -0.03527607361963203)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x105872ed0>: (0.20857586857586857, 0.011758819114335739)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a990>: (0.19086136086136085, -0.008466135458167434)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a810>: (0.23502386502386502, 0.05189620758483027)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a550>: (0.1916072816072816, -0.014349332013854577)
Global model test performance : (0.1401098901098901, -0.05985037406483795)

Comparison with Centralized Training.

# Train model on centralized data:
model_centralized = LogisticRegressionModel(n_features=n_features, classes=classes)
model_centralized.train(train_data, train_labels)
if n_features == 2:
    plot_2D_decision_boundary(model_centralized._model, train_data, train_labels, title = "Benchmark: Logistic regression using Centralized data")
print('Centralized test performance: ' + str(model_centralized.evaluate(test_data, test_labels)))
Centralized test performance: (0.1990731490731491, 0.015698978320458412)

Adding Differential Privacy to the Federated Model.

# Create sampling database:
n_samples = 500
sampling_data, sampling_labels = make_classification(
    n_samples=n_samples, n_features=n_features, n_informative=4,
    n_redundant=0, n_repeated=0, n_classes=n_classes,
    n_clusters_per_class=2, weights=None, flip_y=0.1, class_sep=0.1, random_state=123)   
sample_data = np.hstack((sampling_data, sampling_labels.reshape(-1,1)))
distribution = UniformDistribution(sample_data)

# Sample sensitivity:
n_samples = 300
max_sensitivity, mean_sensitivity = sampler.sample_sensitivity(
    LogisticRegressionSample(n_features=n_features, classes=classes),
    L1SensitivityNorm(), distribution, n=n_samples, gamma=0.05)
print("Max sensitivity from sampling: " + str(max_sensitivity))
print("Mean sensitivity from sampling: " + str(mean_sensitivity))
Max sensitivity from sampling: 1.1229644894348378
Mean sensitivity from sampling: 0.5075662222820022
from shfl.differential_privacy import LaplaceMechanism

params_access_definition = LaplaceMechanism(sensitivity=max_sensitivity, epsilon=0.5)
federated_governmentDP = shfl.federated_government.FederatedGovernment(
    model_builder, federated_data, aggregator, model_params_access=params_access_definition)

federated_governmentDP.run_rounds(n=2, test_data=test_data, test_label=test_labels)
Accuracy round 0
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x105872d90>: (0.17103896103896105, -0.03527607361963203)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x105872ed0>: (0.20857586857586857, 0.011758819114335739)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a990>: (0.19086136086136085, -0.008466135458167434)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a810>: (0.23502386502386502, 0.05189620758483027)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a550>: (0.1916072816072816, -0.014349332013854577)
Global model test performance : (0.18355866355866354, -0.004577822990844416)



Accuracy round 1
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x105872d90>: (0.17103896103896105, -0.03527607361963203)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x105872ed0>: (0.20857586857586857, 0.011758819114335739)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a990>: (0.19086136086136085, -0.008466135458167434)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a810>: (0.23502386502386502, 0.05189620758483027)
Test performance client <shfl.private.federated_operation.FederatedDataNode object at 0x10587a550>: (0.1916072816072816, -0.014349332013854577)
Global model test performance : (0.16416583416583416, -0.02937484308310334)