👉 with all 7 features, a simple vanilla neural network with hidden dimension [64] as regressor, trained 200 epochs.
(kaggle--feedback-prize-ell) PS D:\github\kaggle--feedback-prize-ell> & D:/Users/guido/miniconda3/envs/kaggle--feedback-prize-ell/python.exe d:/github/kaggle--feedback-prize-ell/main.py
MSFTDeBertaV3Config object:
model name: deberta-v3-base
traning device: cuda:0
inference device: cuda:0
model type: nn
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
this torch model will be trained on cuda:0
loading training data...
transforming training data...
using batch transform
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 262/262 [08:34<00:00, 1.96s/it]
training...
model info:
hidden_dims: [64]
Sequential(
(0): Linear(in_features=2282, out_features=64, bias=True)
(1): ReLU()
(2): Linear(in_features=64, out_features=6, bias=True)
(3): ELLActivation()
)
using validation
training completed
testing...
using batch transform
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 130/130 [06:26<00:00, 2.98s/it]
evaluating...
0.47541889625657463
ploting losses...
all done
(kaggle--feedback-prize-ell) PS D:\github\kaggle--feedback-prize-ell>
p.s. feature pipelines
features_pipeline = FeatureUnion([
("unigrams_count", number_of_unigrams_pipe),
("line_breaks_count", number_of_line_breaks_pipe),
("english_score", make_english_score_pipe(fastext_model_path)),
("i_vs_I", i_pipe),
("bad_punctuation", bad_punctuation_pipe),
("tf-idf", tf_idf_pipe),
("deberta_pipe", make_deberta_pipe(deberta_config))
])👉 training log in kaggle, with GPU
/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.5
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
MSFTDeBertaV3Config object:
model name: deberta-v3-base
traning device: cuda:0
inference device: cuda:0
model type: nn
Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
this torch model will be trained on cuda:0
loading training data...
transforming training data...
using batch transform
100%|█████████████████████████████████████████| 262/262 [02:01<00:00, 2.16it/s]
training...
model info:
hidden_dims: [64]
Sequential(
(0): Linear(in_features=2282, out_features=64, bias=True)
(1): ReLU()
(2): Linear(in_features=64, out_features=6, bias=True)
(3): ELLActivation()
)
using validation
training completed
testing...
using batch transform
100%|█████████████████████████████████████████| 130/130 [01:02<00:00, 2.09it/s]
evaluating...
0.4778715845017014
ploting losses...
making submission file...
loading test file from: '/kaggle/input/feedback-prize-english-language-learning/test.csv'
using batch transform
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 6.64it/s]
writing submission to: '/kaggle/working/submission.csv'
all done