Making TensorFlowTransform trainable. #1063

zeahmed · 2018-09-26T22:59:48Z

This PR addressed #951 where user can retrain TensorFlow models using TensorFlowTransform in ML.Net.

The implemented training functionality is an add-on feature on top of current implementation of TensorFlowTransform where model scoring using TensorFlowTransform is untouched.

Current implementation includes

Retraining of un-frozen model (saved with simple_save method in python).
Support for batch training
Serializing retrained model into ML.Net stream

ToDo

[SOLVED] Serializing retrained model into ML.Net stream. It seems like a big challenge right now because TensorFlow C-API does not support serialization of TFSession objects which contains all the retrained parameter.
Turn TensorFlowEstimator from TrivialEstimator to NonTrivialEstimator. This is because TensorFlowTransform now trains model in one of the constructor call where TrainingArguments are passed (Will be done as separate PR Convert TensorFlowEstimator into Non-TrivialEstimator. #1110).
Think about how we can get ride of data copying during creation of batches.

zeahmed · 2018-09-26T23:01:37Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+        private class TensorValueGetter<T> : ITensorValueGetter
+        {
+            private readonly ValueGetter<T> _srcgetter;
+            private readonly List<T> _bufferedData;


List [](start = 29, length = 7)

may be think about making it an array to make it faster. #Resolved

zeahmed · 2018-09-26T23:11:51Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+        [Fact]
+        public void TensorFlowTransformMNISTTemplateTrainingTest()
+        {
+            var model_location = @"E:\Tensorflow\LogisticRegression";


@"E:\Tensorflow\LogisticRegression" [](start = 33, length = 35)

Need to fix this harded coded path... #Resolved

zeahmed · 2018-09-27T01:04:12Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+                trans = new ConcatTransform(env, "Features", "Prediction").Transform(trans);
+                trans = new CopyColumnsTransform(env, ("LabelOriginal", "Label")).Transform(trans);
+
+                var trainer = new LightGbmMulticlassTrainer(env, "Label", "Features");


LightGbmMulticlassTrainer [](start = 34, length = 25)

Note: a trainer on top is actually not a limitation. In Estimator/Transformer world,prediction ipeline will work without adding a trainer at the end. This is just because it being used in direct instantiation case. #Resolved

ericstj · 2018-09-27T16:11:56Z

Serializing retrained model into ML.Net stream. It seems like a big challenge right now because TensorFlow C-API does not support serialization of TFSession objects which contains all the retrained parameter.

Is enough of this state exposed so that we can write serialization on top, or are things internal / not exposed to C-API? At least the Python and JS APIs are able to save so there must be a way, but I can imagine that the exports needed aren't exposed by the TF C-API (or the underlying structure of the objects isn't exposed in the headers). #Closed

zeahmed · 2018-09-27T16:34:39Z

I found out a standard way of doing this through session. I will update the PR with my changes most probably today.

In reply to: 425152137 [](ancestors = 425152137)

zeahmed · 2018-09-27T17:14:18Z

For that I also included some other features from TF#. Your feedback will be valuable on that.

In reply to: 425159496 [](ancestors = 425159496,425152137)

abgoswam · 2018-09-27T17:34:55Z

If we can save the re-trained model in the SavedModel format we will be able to use capabilities for save & restore of SavedModel format that we added in #784

In reply to: 425171940 [](ancestors = 425171940,425159496,425152137)

abgoswam · 2018-09-27T18:17:30Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+                trans = CategoricalTransform.Create(env, trans, "Label");
+                trans = NormalizeTransform.CreateMinMaxNormalizer(env, trans, "Features", "Placeholder");
+
+                var args = new TensorFlowTransform.TrainingArguments()


TrainingArguments [](start = 51, length = 17)

So in this scenario we are re-training the entire graph. this is good place to start the PoC.

Two other patterns for transfer-learning we need to look into :

(a) replace last layer (or last few layers) with a layer that matches shape of "Label" for the new dataset that we want to re-train on.

i.e. re-train the "lr_model" above but instead of the final layer what current has shape [-1,10] I want to re-train on iris dataset (say) which has shape [-1, 5]

(b) support 'freeze_up_to_node=<node_name>' functionality. i.e. freeze weights for nodes that <node_name> depends on, and only fine-tune weights of nodes that depend on <layer_name>

#ByDesign

Thanks @abhishek for pointing out these scenarios. Actually, these are mostly research related scenario and used for experimentation purposes. These can be added on top of what has been implemented so far if required.

BTW, it is no longer a PoC...:)
I would suggest reviewing it as real PR that will go in master.

In reply to: 221026509 [](ancestors = 221026509)

Should we track those scenarios above as separate issues ?

To me they seem more relevant -- esp (a). Otherwise we would have to tell customers that we can only do transfer learning if our label dimension is same as label dimension used in the pre-trained model.

(maybe there might be a simple fix of this -- just want to make sure we keep track of assumptions made in this PR)

In reply to: 221084395 [](ancestors = 221084395,221026509)

Changing the label dimension means changing cost function, prediction function, label input dimension and whatnot. Its hard to identify operations in the graph that are using label and adjust them to new dimensions/settings.
If you have a solution then definitly it would be worth trying...:)

In reply to: 221099796 [](ancestors = 221099796,221084395,221026509)

Ok. I can take a shot at this as a separate issue. We need to document the existing behaviour though i.e. we only support transfer learning on models which have same feature/label dimensions as user data

In reply to: 221102056 [](ancestors = 221102056,221099796,221084395,221026509)

zeahmed · 2018-09-27T21:33:58Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+                    for (int j = 1; j < tfInputShapes[i].NumDimensions; j++)
+                        newShape[j] = tfInputShapes[i][j];
+                    tfInputShapes[i] = new TFShape(newShape);
+                }


Need to bit more shape checking here due to missing dimensions... #Resolved

zeahmed · 2018-09-27T21:36:18Z

Yup, did it the same way.

In reply to: 425178632 [](ancestors = 425178632,425171940,425159496,425152137)

zeahmed · 2018-09-27T22:29:37Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+        ///         })
+        /// </summary>
+        [Fact]
+        public void TensorFlowTransformMNISTTemplateTrainingTest()


TensorFlowTransformMNISTTemplateTrainingTest [](start = 20, length = 44)

Write more tests... #Resolved

abgoswam · 2018-09-27T22:41:44Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+            // Save the model on disk
+            var path = Path.Combine(modelDir, "mlnet_model");
+            Session.GetRunner().AddInput("save/Const", TFTensor.CreateString(Encoding.UTF8.GetBytes(path)))
+                    .AddTarget("save/control_dependency").Run();


Few questions:

is this what's serializing the Session to disk ?

it will save only the parameter weights, not the saved_model.pb graph itself .. am i right ?

Is this a standard technique -- of using nodes with names "save/Const" and "save/control_dependency" to overwrite model weights #Closed

Yes to all three question above.
TensorFlow saves/restores models through these operations on graph.

In reply to: 221098961 [](ancestors = 221098961)

for #3, does TF document this somewhere in code / other language bindings etc.

Just using a combination of strings "save/Const" and and "save/control_dependency" seems so arbitrary :) ..

wonder if this is the same technique used when checkpointing models ?

In reply to: 221102898 [](ancestors = 221102898,221098961)

Yes, this is how TensorFlow saves/restores models to/from disks. Both for checkpointing and simple_save. There is a reason to inject those operation in the graph because TF prefers all operation through graph instead of doing it through python or other language code. I will find a proper documentation for that.

I have made a parameter in TrainingArgument class so that in case user need to change it they can change. However, it is highly unlikely to get changed.

In reply to: 221104033 [](ancestors = 221104033,221102898,221098961)

abgoswam · 2018-09-27T22:52:19Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+        ///
+        /// # Set model weights
+        /// W = tf.Variable(tf.truncated_normal([784, 10]), name = "W")
+        /// b = tf.Variable(tf.constant(0.1, shape=[10], dtype=tf.float32), name = "b")


constant [](start = 31, length = 8)

can we verify that after the fine-tuning we are actually ending up with different values for the parameters ?

e.g. verifying the bias term we get after fine-tuning the model is different from the initialization value (0.1) #Closed

Yes, I will do it.

In reply to: 221100728 [](ancestors = 221100728)

Ivanidzo4ka · 2018-10-01T21:34:27Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

@@ -38,7 +39,7 @@ namespace Microsoft.ML.Transforms
    /// <include file='doc.xml' path='doc/members/member[@name="TensorflowTransform"]/*' />
    public sealed class TensorFlowTransform : ITransformer, ICanSaveModel
    {
-        public sealed class Arguments : TransformInputBase
+        public class Arguments : TransformInputBase


public class Arguments [](start = 8, length = 22)

Can you make it abstract? Also I would suggest to call it ArgumentsBase rather than Arguments, because it would be confusing to have Arguments class which is actually an class with params and this class. #Closed

It can not be made abstract because Arguments object is instantiated for scoring scenario and TrainingArguments is instantiated for training scenario.

In reply to: 221764289 [](ancestors = 221764289)

Why it can't remain sealed if you don't use it as base class?
Also maybe rename it into ScoringArguments?

In reply to: 221768356 [](ancestors = 221768356,221764289)

yaeldekel · 2018-10-03T23:46:33Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+        {
+            float loss = 0;
+            float metric = 0;
+            var runner = Session.GetRunner();


runner [](start = 16, length = 6)

I want to make sure I understand: the runner is tuning the weights of the graph in runner.Run()? and the weights are stored somewhere in Session? #Closed

Yes, the line 451

runner.AddTarget(args.OptimizationOperation);

adds the optimization operation in this runner that updates the weights. Session hold those weights.

In reply to: 222501506 [](ancestors = 222501506)

yaeldekel · 2018-10-03T23:49:29Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+        {
+            if (isVector)
+                return new TensorValueGetterVec<T>(input, colIndex, tfShape);
+            else


else [](start = 12, length = 4)

nit: "else" not needed here. #Closed

yaeldekel · 2018-10-04T15:34:58Z

src/Microsoft.ML.TensorFlow/TensorFlow/Tensorflow.cs

@@ -268,7 +268,7 @@ internal class TFStatus : TFDisposable
    {
        // extern TF_Status * TF_NewStatus ();
        [DllImport(NativeBinding.TensorFlowLibrary)]
-        private static extern unsafe TF_Status TF_NewStatus();
+        internal static extern unsafe TF_Status TF_NewStatus();


internal [](start = 8, length = 8)

Instead of exposing the C-API directly, you can use the C# API defined on top of it (the ctor below). Same for the other C imports below. #Resolved

yaeldekel · 2018-10-04T15:36:39Z

src/Microsoft.ML.TensorFlow/TensorFlow/Tensorflow.cs

@@ -2238,4 +2244,19 @@ public TFTensor AsTensor()
            return shape.AsTensor();
        }
    }
+
+    internal class TFString


TFString [](start = 19, length = 8)

Consider importing these APIs directly in the TFTensor class, this way they can be private. #Resolved

yaeldekel · 2018-10-04T15:50:05Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+            /// The name of the column used as label for training.
+            /// </summary>
+            [Argument(ArgumentType.AtMostOnce, HelpText = "Training labels.", ShortName = "label", SortOrder = 4)]
+            public string LabeLColumn = DefaultColumnNames.Label;


LabeLColumn [](start = 26, length = 11)

This must be identical to the name of the node in the graph corresponding to the label, correct?
If this is the case, then I don't think we should have the default value be "Label". #Resolved

Actually, I think it would make more sense to have two arguments: one for the name of the label column in the data view, and one for the name of the label node in the graph (unless it is possible to find it in the graph without user input?).

In reply to: 222725910 [](ancestors = 222725910)

This is the same way the InputColumns and OutputColumns are treated. Do you think we should treat Label differently? What I can do is make it required.

In reply to: 222726613 [](ancestors = 222726613,222725910)

I don't think we should treat it differently, I think we should also make that change for input/output columns :-).

In reply to: 222754435 [](ancestors = 222754435,222726613,222725910)

I see. I think we already have #769 for tracking this. I can add work for label column as well. Do you agree to do it later?

In reply to: 222841987 [](ancestors = 222841987,222754435,222726613,222725910)

yaeldekel · 2018-10-04T15:54:33Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

@@ -316,7 +695,14 @@ public void Save(ModelSaveContext ctx)
            {
                ctx.SaveBinaryStream("TFSavedModel", w =>
                {
-                    string[] modelFilePaths = Directory.GetFiles(_savedModelPath, "*", SearchOption.AllDirectories);
+                    // only these files need to be saved.
+                    string[] modelFilePaths =


modelFilePaths [](start = 29, length = 14)

Is this a change in the serialization format, or just a more explicit way of writing it? #Resolved

No change in the format. Previously, our assumption was that folder will contain only these files so were writing all files in the folder. Now folder may contain archive files due retraining. So, I am explicitly writing required files.

In reply to: 222727569 [](ancestors = 222727569)

Ivanidzo4ka · 2018-10-04T16:03:03Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+                        if (args.LossOperation != null)
+                            pch.SetHeader(new ProgressHeader("Average Loss"), (e) => e.SetProgress(1, loss / ((cursor.Position + 1) / args.BatchSize)));
+                        if (args.MetricOperation != null)
+                            pch.SetHeader(new ProgressHeader("Average Metric"), (e) => e.SetProgress(2, metric / ((cursor.Position + 1) / args.BatchSize)));


/// A common interface for progress reporting.
/// It is expected that the progress channel interface is used from only one thread.
///
/// Supported workflow:
/// 1) Create the channel via .
/// 2) Call as many times as desired (including 0).
/// Each call to supersedes the previous one.
/// 3) Report checkpoints (0 or more) by calling .
/// 4) Repeat steps 2-3 as often as necessary.
/// 5) Dispose the channel.

see 2).

I think you need to do something like this:

pch.SetHeader(new ProgressHeader(metrics.ToArray() new[] { "Epochs" }, entry => entry.SetProgress(0, epoch, _args.Epochs);

and after each train
if (metric.Count>0)
pch.Checkpoint(double[] avgMetrics);
where metrics and avgMetrics depend on your arguments.

#Closed

Ivanidzo4ka

yaeldekel · 2018-10-04T22:15:11Z

src/Microsoft.ML.TensorFlow/TensorFlow/Tensor.cs

+            // Clear offset table
+            IntPtr dst = TF_TensorData(handle);
+            Marshal.WriteInt64(dst, 0);
+            var status = new TFStatus();


status [](start = 16, length = 6)

using (var status ...) #Resolved

yaeldekel · 2018-10-04T22:15:44Z

src/Microsoft.ML.TensorFlow/TensorFlow/Tensorflow.cs

@@ -290,7 +290,7 @@ public TFStatus() : base(TF_NewStatus())

        // extern void TF_DeleteStatus (TF_Status *);
        [DllImport(NativeBinding.TensorFlowLibrary)]
-        private static extern unsafe void TF_DeleteStatus(TF_Status status);
+        internal static extern unsafe void TF_DeleteStatus(TF_Status status);


internal [](start = 8, length = 8)

Can this one and TF_GetCode go back to being private? #Closed

yaeldekel · 2018-10-05T17:35:42Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+
+        private (int, bool, TFDataType, TFShape) GetInputMetaData(ISchema inputSchema, string columnName, string tfNodeName, int batchSize)
+        {
+            if (!inputSchema.TryGetColumnIndex(columnName, out int inputColIndices))


inputColIndices [](start = 67, length = 15)

inputColIndex #Resolved

yaeldekel · 2018-10-05T17:37:07Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+            var index = inputsForTraining.Length - 1;
+            inputsForTraining[index] = args.TensorFlowLabel;
+            (inputColIndices[index], isInputVector[index], tfInputTypes[index], tfInputShapes[index]) =
+                    GetInputMetaData(inputSchema, args.LabelColumn, inputsForTraining[index], args.BatchSize);


LabelColumn [](start = 55, length = 11)

Check that this is not null. #Resolved

Added that check inCheckTrainingParameters method.

In reply to: 223085996 [](ancestors = 223085996)

yaeldekel · 2018-10-05T17:37:44Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+
+            return (inputColIndices, isInputVector, tfInputType, tfInputShape);
+        }
+        private void TrainCore(Arguments args, string model, IDataView input)


private [](start = 8, length = 7)

Add an empty line. #Resolved

yaeldekel · 2018-10-05T17:38:52Z

src/Microsoft.ML.TensorFlow/TensorflowTransform.cs

+                    var srcTensorGetters = GetTensorValueGetters(cursor, inputColIndices, isInputVector, tfInputTypes, tfInputShapes);
+
+                    float loss=0;
+                    float metric=0;


Ctrl+k+d. #Resolved

yaeldekel · 2018-10-05T17:39:45Z

/// <include file='doc.xml' path='doc/members/member[@name="TensorflowTransform"]/*' />

Please add documentation about this new feature. #Resolved

Refers to: src/Microsoft.ML.TensorFlow/TensorflowTransform.cs:39 in 9c07a53. [](commit_id = 9c07a53, deletion_comment = False)

yaeldekel · 2018-10-05T17:43:03Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+                }, new MultiFileSource(dataPath));
+
+                IDataView trans = CategoricalTransform.Create(env, loader, "OneHotLabel","Label");
+                trans = NormalizeTransform.CreateMinMaxNormalizer(env, trans, "Features", "Placeholder");


CreateMinMaxNormalizer [](start = 43, length = 22)

Why do we add a normalizer here? #Resolved

This is the simple TensorFlow LR model which need normalization otherwise gradients blow out.

In reply to: 223087660 [](ancestors = 223087660)

yaeldekel · 2018-10-05T17:44:25Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+
+                var trainedTfDataView = TensorFlowTransform.Create(env, args, trans);
+                trans = new ConcatTransform(env, "Features", "Prediction").Transform(trainedTfDataView);
+                //trans = new CopyColumnsTransform(env, ("LabelOriginal", "Label")).Transform(trans);


//trans = new CopyColumnsTransform(env, ("LabelOriginal", "Label")).Transform(trans); [](start = 16, length = 85)

Delete. #Closed

…e is error.

into tf_training

zeahmed added 2 commits September 26, 2018 14:45

Making TensorflowTransform trainable.

22d08d6

Merge remote-tracking branch 'upstream/master' into tf_training

1fc443a

zeahmed commented Sep 26, 2018

View reviewed changes

zeahmed requested review from abgoswam, shauheen, ericstj, yaeldekel, markusweimer, TomFinley and Zruty0 September 26, 2018 23:03

zeahmed commented Sep 26, 2018

View reviewed changes

Added model files and script to create model in python.

ef67dd1

zeahmed commented Sep 27, 2018

View reviewed changes

abgoswam reviewed Sep 27, 2018

View reviewed changes

Added functionality to serialized TensorFlow model after retraining.

c1eafa6

zeahmed changed the title ~~WIP: Making TensorFlowTransform trainable.~~ Making TensorFlowTransform trainable. Sep 27, 2018

zeahmed changed the title ~~Making TensorFlowTransform trainable.~~ WIP: Making TensorFlowTransform trainable. Sep 27, 2018

zeahmed commented Sep 27, 2018

View reviewed changes

zeahmed changed the title ~~WIP: Making TensorFlowTransform trainable.~~ Making TensorFlowTransform trainable. Sep 27, 2018

zeahmed commented Sep 27, 2018

View reviewed changes

abgoswam reviewed Sep 27, 2018

View reviewed changes

zeahmed added 2 commits October 1, 2018 14:18

Added comments. Added checking for parameters and types.

07ae61c

Merge remote-tracking branch 'upstream/master' into tf_training

6ec4a5a

Ivanidzo4ka reviewed Oct 1, 2018

View reviewed changes

yaeldekel reviewed Oct 3, 2018

View reviewed changes

Addressed reviewers' comments.

8ea8e9d

yaeldekel reviewed Oct 4, 2018

View reviewed changes

Ivanidzo4ka reviewed Oct 4, 2018

View reviewed changes

Addressed reviewers' comments.

735f7ca

Ivanidzo4ka approved these changes Oct 4, 2018

View reviewed changes

yaeldekel reviewed Oct 4, 2018

View reviewed changes

Addressed reviewers' comments.

9c07a53

yaeldekel reviewed Oct 5, 2018

View reviewed changes

zeahmed added 7 commits October 5, 2018 11:32

Addressed reviewers' comments.

14ec78b

Addressed reviewers' comments.

afb3ca1

Added try-finally in tests so that folders are deleted even when ther…

5a6cc08

…e is error.

Merge conflicts.

4297376

Merge remote-tracking branch 'upstream/master' into tf_training

c0fe791

Merge remote-tracking branch 'upstream/master' into tf_training

0984b0f

Merge branch 'tf_training' of https://github.com/zeahmed/machinelearning

18040ca

into tf_training

zeahmed merged commit d2ed0ad into dotnet:master Oct 8, 2018

zeahmed self-assigned this Oct 9, 2018

ghost locked as resolved and limited conversation to collaborators Mar 28, 2022

Making TensorFlowTransform trainable. #1063

Making TensorFlowTransform trainable. #1063

Conversation

zeahmed commented Sep 26, 2018 • edited Loading

ToDo

zeahmed Sep 26, 2018 • edited Loading

Choose a reason for hiding this comment

zeahmed Sep 26, 2018 • edited Loading

Choose a reason for hiding this comment

zeahmed Sep 27, 2018 • edited Loading

Choose a reason for hiding this comment

ericstj commented Sep 27, 2018 • edited by abgoswam Loading

zeahmed commented Sep 27, 2018

zeahmed commented Sep 27, 2018

abgoswam commented Sep 27, 2018 • edited Loading

abgoswam Sep 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeahmed Sep 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeahmed Sep 27, 2018 • edited Loading

Choose a reason for hiding this comment

zeahmed commented Sep 27, 2018

zeahmed Sep 27, 2018 • edited Loading

Choose a reason for hiding this comment

abgoswam Sep 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abgoswam Sep 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Oct 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaeldekel Oct 3, 2018 • edited Loading

Choose a reason for hiding this comment

zeahmed Oct 4, 2018 • edited Loading

Choose a reason for hiding this comment

yaeldekel Oct 3, 2018 • edited Loading

Choose a reason for hiding this comment

yaeldekel Oct 4, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

yaeldekel Oct 4, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

yaeldekel Oct 4, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeahmed Oct 4, 2018 • edited Loading

Choose a reason for hiding this comment

yaeldekel Oct 4, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Oct 4, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

yaeldekel Oct 4, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

yaeldekel Oct 4, 2018 • edited Loading

Choose a reason for hiding this comment

yaeldekel Oct 5, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

yaeldekel Oct 5, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaeldekel Oct 5, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

yaeldekel Oct 5, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

yaeldekel commented Oct 5, 2018 • edited by zeahmed Loading

yaeldekel Oct 5, 2018 • edited by zeahmed Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeahmed commented Sep 26, 2018 •

edited

Loading

zeahmed Sep 26, 2018 •

edited

Loading

zeahmed Sep 26, 2018 •

edited

Loading

zeahmed Sep 27, 2018 •

edited

Loading

ericstj commented Sep 27, 2018 •

edited by abgoswam

Loading

abgoswam commented Sep 27, 2018 •

edited

Loading

abgoswam Sep 27, 2018 •

edited

Loading

zeahmed Sep 27, 2018 •

edited

Loading

zeahmed Sep 27, 2018 •

edited

Loading

zeahmed Sep 27, 2018 •

edited

Loading

abgoswam Sep 27, 2018 •

edited

Loading

abgoswam Sep 27, 2018 •

edited

Loading

Ivanidzo4ka Oct 1, 2018 •

edited

Loading

yaeldekel Oct 3, 2018 •

edited

Loading

zeahmed Oct 4, 2018 •

edited

Loading

yaeldekel Oct 3, 2018 •

edited

Loading

yaeldekel Oct 4, 2018 •

edited by zeahmed

Loading

yaeldekel Oct 4, 2018 •

edited by zeahmed

Loading

yaeldekel Oct 4, 2018 •

edited by zeahmed

Loading

zeahmed Oct 4, 2018 •

edited

Loading

yaeldekel Oct 4, 2018 •

edited by zeahmed

Loading

Ivanidzo4ka Oct 4, 2018 •

edited

Loading

yaeldekel Oct 4, 2018 •

edited by zeahmed

Loading

yaeldekel Oct 4, 2018 •

edited

Loading

yaeldekel Oct 5, 2018 •

edited by zeahmed

Loading

yaeldekel Oct 5, 2018 •

edited by zeahmed

Loading

yaeldekel Oct 5, 2018 •

edited by zeahmed

Loading

yaeldekel Oct 5, 2018 •

edited by zeahmed

Loading

yaeldekel commented Oct 5, 2018 •

edited by zeahmed

Loading

yaeldekel Oct 5, 2018 •

edited by zeahmed

Loading

yaeldekel Oct 5, 2018 •

edited

Loading