Hotgym predictor, anomaly tests #675

breznak · 2019-09-18T10:16:28Z

adds tests for TM.anomaly and its convergence
WIP adds Predictor (Classifier) to hotgym example, but the use seems to be broken

This PR provides tests for

anomaly scores in different modes, Anomaly Likelihood does not work correctly! #665
testing Predictor/Classifier for SDRClasifier for Prediction? #646 on sine wave data

remove explicit AnomalyLikelihood

but Predictor learn() crashes in Release, however does not in Debug (-> hard to investigate)

breznak

Please review and advise with the issues.

SDRClassifier, Predictor seems broken?
anomaly bindings broken for new setAnomalyMode, but that is not crucial.
improved anomaly tests

breznak · 2019-09-18T10:17:58Z

bindings/py/cpp_src/bindings/algorithms/py_TemporalMemory.cpp

@@ -47,7 +47,7 @@ Example usage:
    TODO
 )");

-	py::enum_<TemporalMemory::ANMode>(m, "ANMode")
+	py::enum_<TemporalMemory::ANMode>(m, "ANMode") //TODO currently htm.bindings.algorithms.ANMode, make ANMode part of algorithms.TemporalMemory


nit, how would it be possible to make enum ANMode exposed as a part of htm.bindings.algorithms.TemporalMemory.ANMode, and not the current ..bindings.algorithms.ANMode ?

breznak · 2019-09-18T10:18:27Z

bindings/py/cpp_src/bindings/algorithms/py_TemporalMemory.cpp

@@ -372,6 +372,8 @@ R"()");
          "Anomaly score updated with each TM::compute() call. "
        );

+	py_HTM.def("setAnomalyMode", &HTM_t::setAnomalyMode);


defined, compiles, but not found in py test?!

breznak · 2019-09-18T10:19:24Z

bindings/py/tests/algorithms/temporal_memory_test.py

+
+    modes = [ANMode.RAW, ANMode.LIKELIHOOD, ANMode.LOGLIKELIHOOD]
+    for mod in modes: #this block test convergence of TM and anomaly score for select mode
+      #FIXME why not visible from bidngings? tm.setAnomalyMode(mod)


setAnomalyMode not found in bindings (see its definition there above)

breznak · 2019-09-18T10:20:01Z

bindings/py/tests/algorithms/temporal_memory_test.py

+      for _ in range(200):
+        inp.addNoise(0.02) #change 2% bits -> 98% overlap => anomaly should ideally be 2%
+        tm.compute(inp, learn=True)
+      self.assertLess(tm.anomaly, 0.08)


new test that TM actually learns and anomalies converge.

src/examples/hotgym/HelloSPTP.cpp

src/htm/algorithms/SDRClassifier.hpp

breznak · 2019-09-18T10:27:51Z

src/htm/algorithms/TemporalMemory.hpp

+      void reset() {
+        anomaly_ = 0.5f;
+	mode_ = ANMode::RAW;
+	//TODO provide anomalyLikelihood_.reset();


when calling setAnomalyMode during a running sequence, the likelihood score would be broken, should reset. But reset is not available now for Likelihood (and I don't intend to implement it in this PR, so it stays broken).

crash was on bad_alloc (memory allocation failed). Our encoding to labels (realToCategory_()) had undeflow which resulted in a huge UInt -> tried to allocate extremely large vector -> failed.

breznak

Figured where the problem was!

breznak · 2019-09-20T01:13:52Z

src/examples/hotgym/HelloSPTP.cpp

+ *  helper to transform (Real) data to categories (UInt) for Classifier/Predictor
+ **/
+UInt realToCategory_(const Real r) {
+  return static_cast<UInt>((r+1.0f /*map sin(x):[-1,1] map it to [0,2]*/)*1000); //precision on 3 dec places


the std::bad_alloc bug (only on Release) was caused here, as the mapping to labels used to underflow (UInt -1), resulting in a huge label, which led to a huge PDF vectors. (not in Debug, because that runs only like 2 steps).

breznak · 2019-09-20T01:16:05Z

src/examples/hotgym/HelloSPTP.cpp

+
+    //Classifier, Predictor
+    tCls.start();
+    pred.learn(e, outTM, { realToCategory_(data) }); //FIXME fails with bad_alloc if label is too large! PDF should use map, instead of a vector


FIXME: this should be resolved in the Predictor. A large label should not crash the program (caused by huge PDF if label is large). See if PDF can use map.

the 4 different step-ahead predictions took a lot of time

add assert to catch whether we learn something.

breznak

Merged the recent progress from PR #667 ,

Predictor crashes for large labels
Predictor is not learning on Hotgym / sine wave data

breznak · 2019-09-20T08:07:51Z

src/examples/hotgym/HelloSPTP.cpp

+      cout << "Cls[0]= " << categoryToReal_(argmax(pred.infer(outTM)[0])) << endl;
+      cout << "Cls[100]= " << categoryToReal_(argmax(pred.infer(outTM)[100])) << endl;
+
+      NTA_CHECK( categoryToReal_(argmax(pred.infer(outTM)[0])) != -1) << "Classifier did not learn"; //FIXME Predictor is not learning, this should be ~ sin(49.99)


FIXME: this is always 0, predictor/classifier is not learning!
Although the unit-tests are passing.
@Thanh-Binh PR #667 should have fixed the symptoms you've described. But my predictor is still failing here.
Is this the same issue as you described, does it replicate your (failing) experiments?

@Thanh-Binh ping, can you please have a look at this PR and the test here? I'm getting 0s from the Classifier inference and not sure why.

this crashes the Predictor. Likely a stack-overflow,

limit to unsingned short (from UInt)

to avoid overflow for large labels

breznak · 2019-09-20T11:18:30Z

I'm considering the following modifications to simplify Predictor, and fix the error with labels:

make inputDimensions a required arg in constructor, as the Classifier can operate on just fixed SDR.size. Removes the if-condition from learn()
make optional constructor arg vector<UInt> fixedLabels, this would allow us to skip the logic (in learn) for adding new labels. Many classification tasks have fixed labels (supervised do).
make internal weights_[i]a map (from vector). This is a bit reverting back to history. Using vector forces continuous labels. Alternative is using label indices, as intended. But for unknown/variable labels (unsupervised, ie. encoder's bucket idx) this is more complicated. @ctrl-z-9000-times what's your opinion on this move? (cont. label indices, vs. non-cont. labels) This is in Classifier: use map to allow sparse categories WIP #680 WIP only.

needed to specify the argument for the signiture to be correct

breznak added 3 commits September 18, 2019 10:05

TM.anomaly: add python convergence tests, setAnomalyMode

5585038

Hotgym: use TM.anomaly with different modes

d885653

remove explicit AnomalyLikelihood

Hotgym: use SDRClassifier, Predictor WIP

432f98a

but Predictor learn() crashes in Release, however does not in Debug (-> hard to investigate)

breznak added tests example labels Sep 18, 2019

breznak self-assigned this Sep 18, 2019

breznak commented Sep 18, 2019

View reviewed changes

breznak requested review from ctrl-z-9000-times and dkeeney September 18, 2019 12:22

This was referenced Sep 18, 2019

SDRClasifier for Prediction? #646

Closed

SDRClassifier: fix precision by using Real64 for PDF #667

Merged

breznak added 2 commits September 20, 2019 02:06

Merge branch 'master_community' into hotgym_predictor

9f015f1

Hotgym: debugged crash with Predictor

97023a8

crash was on bad_alloc (memory allocation failed). Our encoding to labels (realToCategory_()) had undeflow which resulted in a huge UInt -> tried to allocate extremely large vector -> failed.

breznak commented Sep 20, 2019

View reviewed changes

breznak added 3 commits September 20, 2019 09:37

Merge branch 'master_community' into hotgym_predictor

ae192bd

Hotgym: fewer steps in Predictor, for faster runtime

0987941

the 4 different step-ahead predictions took a lot of time

Hotgym: Predictor not learning, still zero

0e1cc14

add assert to catch whether we learn something.

breznak commented Sep 20, 2019

View reviewed changes

breznak added 3 commits September 20, 2019 10:25

Predictor: add reproducible test for large labels

6c7678e

this crashes the Predictor. Likely a stack-overflow,

Predictor: use type StepsAheadT for steps_

285490b

limit to unsingned short (from UInt)

Classifier: numCategories_ is size_t

06f9701

to avoid overflow for large labels

breznak added 3 commits November 6, 2019 14:05

Merge branch 'master_community' into hotgym_predictor

4bed29d

fix bindings for setAnomalyMode

f80a765

needed to specify the argument for the signiture to be correct

progress

865f7e0

Zbysekz closed this Jun 26, 2020

Zbysekz deleted the hotgym_predictor branch June 26, 2020 06:58

breznak restored the hotgym_predictor branch June 26, 2020 07:13

breznak reopened this Jun 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hotgym predictor, anomaly tests #675

Hotgym predictor, anomaly tests #675

breznak commented Sep 18, 2019 •

edited

Loading

breznak left a comment

breznak Sep 18, 2019

breznak Sep 18, 2019

breznak Sep 18, 2019

breznak Sep 18, 2019

breznak Sep 18, 2019

breznak left a comment

breznak Sep 20, 2019

breznak Sep 20, 2019

breznak left a comment

breznak Sep 20, 2019 •

edited

Loading

breznak Nov 7, 2019

breznak commented Sep 20, 2019

Hotgym predictor, anomaly tests #675

Are you sure you want to change the base?

Hotgym predictor, anomaly tests #675

Conversation

breznak commented Sep 18, 2019 • edited Loading

breznak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breznak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breznak left a comment

Choose a reason for hiding this comment

breznak Sep 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breznak commented Sep 20, 2019

breznak commented Sep 18, 2019 •

edited

Loading

breznak Sep 20, 2019 •

edited

Loading