This repo collect the models for response selection and merge them into one framework.
- SMN: Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots. pdf
- DAM: Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network. pdf
- IOI: One Time of Interaction May Not Be Enough: Go Deep with an Interaction-over-Interaction Network for Response Selection in Dialogues. pdf
- ESIM: Sequential Matching Model for End-to-end Multi-turn Response Selection. pdf
- MSN: Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots. pdf
Note: I am still working on reproduce the results of MSN. The results pf MSN is still quite low in this repo. Welcome to pull requests to optimize MSN in this repo.
- models are all copied from their official released code.
- The dataset is from MSN.
- The data processing and code structure is from DAM.
- python 3
- tensorflow 1.10
- scipy
- sklearn
- The dataset is from the data released by MSN.
- The dataset can be downloaded from
cd utils
python parse_data.py ../ResponseSelection/ubuntu_data ../data/ubuntu_write
python compose_data.py ../data/ubuntu_write
The configuration is located in main.py
.
python main.py ubuntu train SMN
CHECKPOINT_NAME=output/**
python main.py ubuntu test SMN CHECKPOINT_NAME
Official results from Leaderboards:
Model | R_2@1 | R_10@1 | R_10@2 | R_10@5 |
---|---|---|---|---|
SMN | 0.926 | 0.726 | 0.847 | 0.961 |
DAM | 0.938 | 0.767 | 0.874 | 0.969 |
ESIM | - | 0.796 | 0.894 | 0.975 |
IoI | 0.947 | 0.796 | 0.894 | 0.974 |
MSN | - | 0.800 | 0.899 | 0.978 |
Model | MAP | MRR | P@1 | R_10@1 | R_10@2 | R_10@5 |
---|---|---|---|---|---|---|
SMN | 0.529 | 0.569 | 0.397 | 0.233 | 0.396 | 0.724 |
DAM | 0.550 | 0.601 | 0.427 | 0.254 | 0.410 | 0.757 |
ESIM | - | - | - | - | - | - |
IoI | 0.573 | 0.621 | 0.444 | 0.269 | 0.451 | 0.786 |
MSN | 0.587 | 0.632 | 0.470 | 0.295 | 0.452 | 0.788 |
Model | R_10@1 | R_10@2 | R_10@5 |
---|---|---|---|
SMN | 0.453 | 0.654 | 0.886 |
ESIM | 0.570 | 0.767 | 0.948 |
IoI | 0.563 | 0.768 | 0.950 |
MSN | 0.606 | 0.770 | 0.937 |
Model | R_2@1 | R_10@1 | R_10@2 | R_10@5 |
---|---|---|---|---|
SMN | 0.944 | 0.782 | 0.885 | 0.973 |
DAM | 0.947 | 0.790 | 0.890 | 0.975 |
ESIM | 0.948 | 0.786 | 0.893 | 0.977 |
IoI | 0.954 | 0.814 | 0.904 | 0.978 |
Model | MAP | MRR | P@1 | R_10@1 | R_10@2 | R_10@5 |
---|---|---|---|---|---|---|
SMN | 0.550 | 0.589 | 0.407 | 0.253 | 0.411 | 0.769 |
DAM | 0.562 | 0.607 | 0.428 | 0.256 | 0.427 | 0.784 |
ESIM | 0.569 | 0.614 | 0.437 | 0.270 | 0.441 | 0.772 |
IoI | 0.575 | 0.622 | 0.446 | 0.266 | 0.451 | 0.796 |
Model | R_10@1 | R_10@2 | R_10@5 |
---|---|---|---|
SMN | 0.530 | 0.703 | 0.928 |
DAM | 0.595 | 0.786 | 0.958 |
ESIM | 0.551 | 0.749 | 0.945 |
IoI | 0.601 | 0.817 | 0.960 |
- Upgrade tendorflow to newer version
- Optimizing MSN
- For ESIM, the data processing is not standard from their original paper.
We refer theese papers and repos to build this repo:
- SMN Code: https://github.com/MarkWuNLP/MultiTurnResponseSelection
- DAM Code: https://github.com/baidu/Dialogue
- IOI Code: https://github.com/chongyangtao/IOI
- ESIM Code: https://github.com/alibaba/esim-response-selection
- MSN Code: https://github.com/chunyuanY/Dialogue
I am Zhenxin Fu from Peking University.