Service keeps on restarting

Forum
1

Product/components used and version/fix level:

Cumulocity Production

Detailed explanation of the problem:

Hi, I have this microservice that is using PostgreSQL as database. I deployed it on cumulocity with the configuration shown below. The service is keep on restarting and i am getting these logs. Can any one suggest what to do?


  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.7.0)

2024-11-06 08:33:02.289  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2024-11-06 08:33:02.342  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 43 ms. Found 1 JPA repository interfaces.
2024-11-06 08:33:02.971  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 80 (http)
2024-11-06 08:33:02.982  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-11-06 08:33:02.983  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.63]
2024-11-06 08:33:03.060  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-11-06 08:33:03.061  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1625 ms
2024-11-06 08:33:03.445  INFO 1 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
2024-11-06 08:33:03.498  INFO 1 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 5.6.9.Final
2024-11-06 08:33:03.646  INFO 1 --- [           main] o.hibernate.annotations.common.Version   : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2024-11-06 08:33:03.749  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2024-11-06 08:33:04.087  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2024-11-06 08:33:04.112  INFO 1 --- [           main] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect
2024-11-06 08:33:04.951  INFO 1 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2024-11-06 08:33:04.958  INFO 1 --- [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2024-11-06 08:33:05.301  WARN 1 --- [           main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2024-11-06 08:33:05.765  WARN 1 --- [           main] .s.s.UserDetailsServiceAutoConfiguration : 

Using generated security password: some pass

This generated password is for development use only. Your security configuration must be updated before running your application in production.

2024-11-06 08:33:05.931  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2024-11-06 08:33:05.946  INFO 1 --- [           main] o.s.s.web.DefaultSecurityFilterChain     : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@46dcbeab, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@36e43829, org.springframework.security.web.context.SecurityContextPersistenceFilter@5a545b0f, org.springframework.security.web.header.HeaderWriterFilter@71ed560f, org.springframework.web.filter.CorsFilter@152c4495, org.springframework.security.web.csrf.CsrfFilter@4d84049a, org.springframework.security.web.authentication.logout.LogoutFilter@430b2699, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@2404b5a, org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter@3ff54f3d, org.springframework.security.web.authentication.ui.DefaultLogoutPageGeneratingFilter@5ec1963c, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@31b82e0f, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@124dac75, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@315105f, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@7dcb42a9, org.springframework.security.web.session.SessionManagementFilter@58253c57, org.springframework.security.web.access.ExceptionTranslationFilter@383caf89, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@696db620]
2024-11-06 08:33:06.027  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 80 (http) with context path ''
: Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-11-06 08:33:30.990  INFO 1 --- [p-nio-80-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2024-11-06 08:33:30.992  INFO 1 --- [p-nio-80-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 2 ms
2024-11-06 08:33:58.210  INFO 1 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2024-11-06 08:33:58.216  INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
2024-11-06 08:33:58.287  INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.

Configuration:

{
  "apiVersion": "1",
  "version": "1.0.4",
  "provider": {
    "name": "my-service"
  },
  "isolation": "MULTI_TENANT",
  "resources": {
    "cpu": "8",
    "memory": "4G"
  },
  "requiredRoles": [
    "ROLE_INVENTORY_READ",
    "ROLE_INVENTORY_ADMIN",
    "ROLE_MEASUREMENT_READ",
    "ROLE_MEASUREMENT_ADMIN"
  ],
  "roles": [],
  "livenessProbe": {
    "httpGet": {
      "path": "/health",
      "port": 80
    },
    "initialDelaySeconds": 30,
    "periodSeconds": 10
  },
  "readinessProbe": {
    "httpGet": {
      "path": "/health",
      "port": 80
    },
    "initialDelaySeconds": 30,
    "periodSeconds": 10
  }
}
2

First of all you should use "apiVersion": "2"
Secondly you should check if your health endpoint is giving a response as expected (200).
Also I would increase the initialDelaySeconds to 60 for the livenessProbe because it can be that the health endpoint isn’t available within that time frame.

3

Hi Stefan,

Can you please explain the difference between api version 1 and 2. For your other suggestions, i get this as the response from the service when i hit my custom endpoints. Also, even if i get access to heatlh endpoint i dont think i will get 200 because in my status i could see 1 unhealthy instance. Refer the image below

untitled
untitled1462×741 75.8 KB

{
    "error": "Microservice/Bad gateway",
    "message": "Microservice not available Connection refused : Connection for request id already closed",
    "info": "https://cumulocity.com/guides/users-guide/getting-started/"
4

Hi Stefan,

Update: I modified the configuration with this but still getting the same error.

Configuration:

{
  "apiVersion": "2",
  "version": "1.0.6",
  "provider": {
    "name": "my-service"
  },
  "isolation": "MULTI_TENANT",
  "resources": {
    "cpu": "8",
    "memory": "4G"
  },
  "requiredRoles": [
    "ROLE_INVENTORY_READ",
    "ROLE_INVENTORY_ADMIN",
    "ROLE_MEASUREMENT_READ",
    "ROLE_MEASUREMENT_ADMIN"
  ],
  "roles": [],
  "livenessProbe": {
    "httpGet": {
      "path": "/health",
      "port": 80
    },
    "initialDelaySeconds": 60,
    "periodSeconds": 10
  },
  "readinessProbe": {
    "httpGet": {
      "path": "/health",
      "port": 80
    },
    "initialDelaySeconds": 30,
    "periodSeconds": 10
  }
}

Error:

{
    "error": "Microservice/Bad gateway",
    "message": "Microservice not available Connection refused : Connection for request id already closed",
    "info": "https://cumulocity.com/guides/users-guide/getting-started/"
}
5

ApiVersion 2 is the current version which should be used as version 1 is deprecated:

It is unhealthy because of the livenessProbe configuration that if the health point is not reachable sets the state to “unhealthy” and tries to restart the microservice in a loop.

For testing purpose you can remove that configuration from your manifest and deploy the microservice again. If it is up and running you should be able to reach your health endpoint. If you can’t there is something wrong with your microservice implementation not running as intended.