[Python] Allowing custom objective / metric function with "objective" and "metric" parameters

[daanklijn]

Summary

It would be nice if one could register custom objective and loss functions, so that these can be passed into the LightGBM's train function via the param argument.

Motivation

I believe that this would be a nice feature as this allows for easier hyperparameter tuning. Currently one would need to write logic that retrieves the custom function based on the name in a config file and pass those into the train function's feval and fobj arguments. It would be much easier and cleaner if one could just register these custom functions s.t. they can be passed in like the standard objective functions and eval metrics.

Description

Currently one would do something like this to use custom objective functions and evaluation metrics:

import lightgbm as lgb
from custom import custom_objective, custom_metric

params = {}
gbm = lgb.train(params=params,
                train_set=train_set,
                valid_sets=eval_set,
                fobj=custom_objective,
                feval=custom_metric
                )

I would propose to also allow something similar to this:

import lightgbm as lgb
from custom import custom_objective, custom_metric

lgb.register_metric(name="custom_metric", function=custom_metric)
lgb.register_objective(name="custom_objective", function=custom_objective)

params = {
    'objective': 'custom_objective',
    'metric': ['custom_metric'],
}

gbm = lgb.train(params=params,
                train_set=train_set,
                valid_sets=eval_set
                )

[guolinke]

thanks @daanklijn
I think this is quite easy to implement.
@shiyu1994 could you help with this when you have time?

[shiyu1994]

thanks @daanklijn
I think this is quite easy to implement.
@shiyu1994 could you help with this when you have time?

Sure.

[StrikerRUS]

@daanklijn Will passing callable metric/loss in standard params will suit your needs?
We have something similar for sklearn API and I personally find it more convenient than registering callables.

LightGBM/examples/python-guide/sklearn_example.py

Line 65 in 3c0e12d

eval_metric=[rmsle, rae],

LightGBM/python-package/lightgbm/sklearn.py

Lines 519 to 521 in 78d31d9

	# Separate built-in from callable evaluation metrics
	eval_metrics_callable = [_EvalFunctionWrapper(f) for f in eval_metric_list if callable(f)]
	eval_metrics_builtin = [m for m in eval_metric_list if isinstance(m, str)]

[StrikerRUS]

Closed in favor of being in #2302. We decided to keep all feature requests in one place.

Welcome to contribute this feature! Please re-open this issue (or post a comment if you are not a topic starter) if you are actively working on implementing this feature.

[TremaMiguel]

@StrikerRUS , @jameslamb I'm open to develop this

I will consider starting first with register_objective because it is only a callable or None.

I think a similar logic as with Loggers can be used, that is, first register the metric (example register_logger) and then making the callable available through a function (example _log_warning) _retrieve_objective, this function either return a callable or None.

Inside lgb.train() if fobj is not provided check for a previous register callable, reference

fobj = fobj or _retrieve_objective()  # either passed argument or registered callable

if fobj is not None:
    for obj_alias in _ConfigAliases.get("objective"):
        params.pop(obj_alias, None)
    params['objective'] = 'none'

what are your thoughts?

[StrikerRUS]

@TremaMiguel Sorry, I'm not 100% sold on registering custom functions. I believe that allowing to pass callables to objective/metric argument is more user friendly and is already implemented in the sklearn wrapper (#3244 (comment)). Adding one more way to pass custom functions will add a lot of maintenance burden and may confuse users by number of possibilities.

[jameslamb]

I agree completely with @StrikerRUS

[TremaMiguel]

Agree I think registering custom callables could bring some unnecessary logic. I've found this issue available on #2302, should it be marked as completed or removed from the list?

[StrikerRUS]

I believe we can wait for some other opinions. Or this issue can be transformed into feature request for allowing custom callables via objective/metric parameter for all Python interfaces.

[jameslamb]

I agree with @StrikerRUS , I'd support passing callables to parameter objective or metric, and don't support adding something like a lgb.register_metric() function.

For more inspiration, the R package also supports passing a mix of strings and custom functions:

LightGBM/R-package/tests/testthat/test_basic.R

Lines 1412 to 1429 in d31346f

	bst <- lgb.train(
	params = list(
	objective = "regression"
	, metric = "None"
	, verbose = VERBOSITY
	)
	, data = DTRAIN_RANDOM_REGRESSION
	, nrounds = nrounds
	, valids = list(
	"valid1" = DVALID_RANDOM_REGRESSION
	)
	, eval = list(
	.increasing_metric
	, "rmse"
	, .constant_metric
	, "l2"
	)
	)

[shiyu1994]

I agree with @jameslamb @StrikerRUS. Allowing passing customized function with parameter "objective" or "metric" can also fulfill the need in the motivation of this feature request. I'll change the description in #2302.

[TremaMiguel]

@jameslamb I'm out of context, as I understand fobj and feval already accept custom callables, what's different with @StrikerRUS proposal?

[jameslamb]

what's different with @StrikerRUS proposal

We are saying we'd support a pull request making it possible to do something like the following:

bst = lgb.train(
    ...,
    params = {
        "metric": ["auc", _some_function_i_wrote],
    }
)

And the same for objective.

Like the code linked in #3244 (comment).

[TremaMiguel]

@jameslamb @StrikerRUS so basically to achieve

bst = lgb.train(
    ...,
    params = {
        "objective": "regression" # or custom function  "objective":custom_objective_function
        "metric": ["auc", _some_function_i_wrote],
    }
)

one needs to get the following done

1. Check if objective is in params and assigned it to fobj like the R implementation. This will be passed to Booster.update()
2. Check if metric is in params and pass it to Booster.eval_train() and Booster.eval_valid()
3. Add support for multiple metric values like the R implementation, creating a list of eval_functions

is there anything missing?

[StrikerRUS]

I think we can remove fobj and feval dedicated arguments after implementing this feature.

[StrikerRUS]

Firstly I guess we can simply transfer this logic

LightGBM/python-package/lightgbm/sklearn.py

Lines 668 to 675 in 01568cf

	# Separate built-in from callable evaluation metrics
	eval_metrics_callable = [_EvalFunctionWrapper(f) for f in eval_metric_list if callable(f)]
	eval_metrics_builtin = [m for m in eval_metric_list if isinstance(m, str)]
	# concatenate metric from params (or default if not provided in params) and eval_metric
	params['metric'] = [params['metric']] if isinstance(params['metric'], (str, type(None))) else params['metric']
	params['metric'] = [e for e in eval_metrics_builtin if e not in params['metric']] + params['metric']
	params['metric'] = [metric for metric in params['metric'] if metric is not None]

into basic.py

[TremaMiguel]

@StrikerRUS Ohh I see, I think this one is kind of easy, one just need to handle:

• metrics with the code you provide, ignoring feval if params has a metric key.
• handle fobj if callable or string with

# objective is callable
f_obj = params['objective']
if callable(f_obj):
    for obj_alias in _ConfigAliases.get("objective"):
        params.pop(obj_alias, None)
    fobj = params['objective']
    params['objective'] = None
elif isinstance(f_obj, str):
    for obj_alias in _ConfigAliases.get("objective"):
        params.pop(obj_alias, None)

and no need to add a list of eval_functions cause Booster.__inner_eval supports it.

P.D: I think there's an improvement with docs regarding Booster eval, eval_train and eval_valid methods #5046