Skip to content

Instantly share code, notes, and snippets.

@rmoff

rmoff/smt.md

Created Jan 24, 2020
Embed
What would you like to do?
Kafka Connect - IllegalArgumentException: Invalid decimal scale: 127

From Mayank Patel on http://cnfl.io/slack

Problem:

org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:284)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:309)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:234)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Invalid decimal scale: 127 (greater than precision: 64)
	at org.apache.avro.LogicalTypes$Decimal.validate(LogicalTypes.java:217)
	at org.apache.avro.LogicalType.addToSchema(LogicalType.java:70)
	at org.apache.avro.LogicalTypes$Decimal.addToSchema(LogicalTypes.java:182)
	at io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:944)
	at io.confluent.connect.avro.AvroData.addAvroRecordField(AvroData.java:1059)
	at io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:900)
	at io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:732)
	at io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:726)
	at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:365)
	at io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:80)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:284)
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
	... 11 more

Fix:

SMT that I ended up adding to my config I am posting when calling connector through API end point (So basically I am changing format for ID and Amount):

"transforms": "Cast",
"transforms.Cast.type": "org.apache.kafka.connect.transforms.Cast$Value",
"transforms.Cast.spec": "ID:int32,AMOUNT:float64"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment