
############################## Experiment details ##############################

dataset: femnist
dataset_dir: ../data/femnist/
output_dir: output/debug_memory_leak/rsvd_r16_cnn-2022-08-07_15:41:49/
model: cnn
num_rounds: 100
eval_every: 10
ServerType: <class 'nla_cnn_exp.nla_cnn.<locals>.make_comp_server_client.<locals>.CompServer'>
client_types: [(<class 'nla_cnn_exp.nla_cnn.<locals>.make_comp_server_client.<locals>.CompClient'>, 100)]
clients_per_round: 10
client_lr: 0.06
batch_size: 64
seed: 0
use_val_set: False
num_epochs: 10
gpus_per_client_cluster: 1
num_client_clusters: 2
save_model: False

############################## Simulation ##############################

Spawning 2 ClientClusters using cuda device (this may take a while)
100 total clients: 100 CompClients

>>> Training Accuracy: Round 0 <<<
+------------+------------+-------------------+-------------------+
|    average |     median |   10th percentile |   90th percentile |
|------------+------------+-------------------+-------------------|
| 0.00254635 | 0.00292398 |                 0 |        0.00377645 |
+------------+------------+-------------------+-------------------+
>>> Test Accuracy: Round 0 <<<
+------------+----------+-------------------+-------------------+
|    average |   median |   10th percentile |   90th percentile |
|------------+----------+-------------------+-------------------|
| 0.00330761 |        0 |                 0 |         0.0239286 |
+------------+----------+-------------------+-------------------+
--- Round 1 of 100: Training 10 clients: 10 CompClients ---
--- Round 2 of 100: Training 10 clients: 10 CompClients ---
--- Round 3 of 100: Training 10 clients: 10 CompClients ---
--- Round 4 of 100: Training 10 clients: 10 CompClients ---
--- Round 5 of 100: Training 10 clients: 10 CompClients ---
--- Round 6 of 100: Training 10 clients: 10 CompClients ---
--- Round 7 of 100: Training 10 clients: 10 CompClients ---
--- Round 8 of 100: Training 10 clients: 10 CompClients ---
--- Round 9 of 100: Training 10 clients: 10 CompClients ---
--- Round 10 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 10 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.549452 | 0.560664 |          0.401981 |          0.672291 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 10 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.549063 | 0.556804 |          0.359714 |          0.704545 |
+-----------+----------+-------------------+-------------------+
--- Round 11 of 100: Training 10 clients: 10 CompClients ---
--- Round 12 of 100: Training 10 clients: 10 CompClients ---
--- Round 13 of 100: Training 10 clients: 10 CompClients ---
--- Round 14 of 100: Training 10 clients: 10 CompClients ---
--- Round 15 of 100: Training 10 clients: 10 CompClients ---
--- Round 16 of 100: Training 10 clients: 10 CompClients ---
--- Round 17 of 100: Training 10 clients: 10 CompClients ---
--- Round 18 of 100: Training 10 clients: 10 CompClients ---
--- Round 19 of 100: Training 10 clients: 10 CompClients ---
--- Round 20 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 20 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.694283 | 0.712923 |           0.53542 |          0.793803 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 20 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.696527 | 0.702703 |          0.486903 |          0.854007 |
+-----------+----------+-------------------+-------------------+
--- Round 21 of 100: Training 10 clients: 10 CompClients ---
--- Round 22 of 100: Training 10 clients: 10 CompClients ---
--- Round 23 of 100: Training 10 clients: 10 CompClients ---
--- Round 24 of 100: Training 10 clients: 10 CompClients ---
--- Round 25 of 100: Training 10 clients: 10 CompClients ---
--- Round 26 of 100: Training 10 clients: 10 CompClients ---
--- Round 27 of 100: Training 10 clients: 10 CompClients ---
--- Round 28 of 100: Training 10 clients: 10 CompClients ---
--- Round 29 of 100: Training 10 clients: 10 CompClients ---
--- Round 30 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 30 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.731671 | 0.742892 |          0.622001 |          0.826792 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 30 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.734013 | 0.746795 |          0.551914 |          0.872711 |
+-----------+----------+-------------------+-------------------+
--- Round 31 of 100: Training 10 clients: 10 CompClients ---
--- Round 32 of 100: Training 10 clients: 10 CompClients ---
--- Round 33 of 100: Training 10 clients: 10 CompClients ---
--- Round 34 of 100: Training 10 clients: 10 CompClients ---
--- Round 35 of 100: Training 10 clients: 10 CompClients ---
--- Round 36 of 100: Training 10 clients: 10 CompClients ---
--- Round 37 of 100: Training 10 clients: 10 CompClients ---
--- Round 38 of 100: Training 10 clients: 10 CompClients ---
--- Round 39 of 100: Training 10 clients: 10 CompClients ---
--- Round 40 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 40 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.733752 | 0.749135 |          0.595916 |          0.836963 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 40 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.732635 | 0.742396 |          0.527753 |          0.882754 |
+-----------+----------+-------------------+-------------------+
--- Round 41 of 100: Training 10 clients: 10 CompClients ---
--- Round 42 of 100: Training 10 clients: 10 CompClients ---
--- Round 43 of 100: Training 10 clients: 10 CompClients ---
--- Round 44 of 100: Training 10 clients: 10 CompClients ---
--- Round 45 of 100: Training 10 clients: 10 CompClients ---
--- Round 46 of 100: Training 10 clients: 10 CompClients ---
--- Round 47 of 100: Training 10 clients: 10 CompClients ---
--- Round 48 of 100: Training 10 clients: 10 CompClients ---
--- Round 49 of 100: Training 10 clients: 10 CompClients ---
--- Round 50 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 50 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.786728 | 0.798632 |          0.680441 |          0.876804 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 50 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.781974 | 0.806004 |          0.608824 |          0.900244 |
+-----------+----------+-------------------+-------------------+
--- Round 51 of 100: Training 10 clients: 10 CompClients ---
--- Round 52 of 100: Training 10 clients: 10 CompClients ---
--- Round 53 of 100: Training 10 clients: 10 CompClients ---
--- Round 54 of 100: Training 10 clients: 10 CompClients ---
--- Round 55 of 100: Training 10 clients: 10 CompClients ---
--- Round 56 of 100: Training 10 clients: 10 CompClients ---
--- Round 57 of 100: Training 10 clients: 10 CompClients ---
--- Round 58 of 100: Training 10 clients: 10 CompClients ---
--- Round 59 of 100: Training 10 clients: 10 CompClients ---
--- Round 60 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 60 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.794119 | 0.811445 |          0.692977 |          0.887317 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 60 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.782249 | 0.800166 |          0.641729 |          0.896897 |
+-----------+----------+-------------------+-------------------+
--- Round 61 of 100: Training 10 clients: 10 CompClients ---
--- Round 62 of 100: Training 10 clients: 10 CompClients ---
--- Round 63 of 100: Training 10 clients: 10 CompClients ---
--- Round 64 of 100: Training 10 clients: 10 CompClients ---
--- Round 65 of 100: Training 10 clients: 10 CompClients ---
--- Round 66 of 100: Training 10 clients: 10 CompClients ---
--- Round 67 of 100: Training 10 clients: 10 CompClients ---
--- Round 68 of 100: Training 10 clients: 10 CompClients ---
--- Round 69 of 100: Training 10 clients: 10 CompClients ---
--- Round 70 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 70 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|    0.8098 | 0.821062 |          0.709731 |           0.89358 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 70 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.792448 | 0.811655 |          0.641729 |               0.9 |
+-----------+----------+-------------------+-------------------+
--- Round 71 of 100: Training 10 clients: 10 CompClients ---
--- Round 72 of 100: Training 10 clients: 10 CompClients ---
--- Round 73 of 100: Training 10 clients: 10 CompClients ---
--- Round 74 of 100: Training 10 clients: 10 CompClients ---
--- Round 75 of 100: Training 10 clients: 10 CompClients ---
--- Round 76 of 100: Training 10 clients: 10 CompClients ---
--- Round 77 of 100: Training 10 clients: 10 CompClients ---
--- Round 78 of 100: Training 10 clients: 10 CompClients ---
--- Round 79 of 100: Training 10 clients: 10 CompClients ---
--- Round 80 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 80 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.820514 | 0.835063 |          0.720108 |          0.903661 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 80 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.800717 | 0.814871 |          0.664706 |          0.904911 |
+-----------+----------+-------------------+-------------------+
--- Round 81 of 100: Training 10 clients: 10 CompClients ---
--- Round 82 of 100: Training 10 clients: 10 CompClients ---
--- Round 83 of 100: Training 10 clients: 10 CompClients ---
--- Round 84 of 100: Training 10 clients: 10 CompClients ---
--- Round 85 of 100: Training 10 clients: 10 CompClients ---
--- Round 86 of 100: Training 10 clients: 10 CompClients ---
--- Round 87 of 100: Training 10 clients: 10 CompClients ---
--- Round 88 of 100: Training 10 clients: 10 CompClients ---
--- Round 89 of 100: Training 10 clients: 10 CompClients ---
--- Round 90 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 90 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.826973 | 0.835794 |          0.717932 |          0.905135 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 90 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.803749 | 0.811655 |          0.666667 |          0.907933 |
+-----------+----------+-------------------+-------------------+
--- Round 91 of 100: Training 10 clients: 10 CompClients ---
--- Round 92 of 100: Training 10 clients: 10 CompClients ---
--- Round 93 of 100: Training 10 clients: 10 CompClients ---
--- Round 94 of 100: Training 10 clients: 10 CompClients ---
--- Round 95 of 100: Training 10 clients: 10 CompClients ---
--- Round 96 of 100: Training 10 clients: 10 CompClients ---
--- Round 97 of 100: Training 10 clients: 10 CompClients ---
--- Round 98 of 100: Training 10 clients: 10 CompClients ---
--- Round 99 of 100: Training 10 clients: 10 CompClients ---
--- Round 100 of 100: Training 10 clients: 10 CompClients ---
>>> Training Accuracy: Round 100 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.829767 | 0.845407 |          0.721283 |          0.905639 |
+-----------+----------+-------------------+-------------------+
>>> Test Accuracy: Round 100 <<<
+-----------+----------+-------------------+-------------------+
|   average |   median |   10th percentile |   90th percentile |
|-----------+----------+-------------------+-------------------|
|  0.807883 | 0.827134 |          0.665385 |          0.921447 |
+-----------+----------+-------------------+-------------------+

############################## Post-Simulation ##############################

Total Experiment time: 0:10:38.749707
