8.1.2 - FusionHub high CPU

We just set up a FusionHub solo on behalf of a customer hosted in Vultr (so KVM platform I guess?) and upgraded it to 8.1.2 but we noticed that the CPU was sitting at 30-40% utilisation with no peers connected and no traffic being passed.

Downgrading the FH to 8.1.1 and the CPU utilisation is down to 1% at idle.

Has anybody else seen this behaviour?

1 Like

Not quite as high, but yes much higher average CPU load:

1 Like

Same here.

@Paul_Mossip and I seem to have pushed the 8.1.2 update on the exact same day. :slight_smile:

1 Like

My FusionHub CPU here is also at 25% hosted on Vultr. I didn’t notice until I went and looked at it. I ran 8.1.2 4998 before running 8.1.2 5005 and it appears that the CPU problem exists in both versions. I don’t have logs going back to before 4998, so I don’t know which version it started under. All I can say is that my idle CPU level used to be 4%

2 Likes

Did a test last night on our own FH hosted in our VMware environment and this is the same behaviour, after rebooting into 8.1.2 the CPU utilisation is reported at an average of 30-35% in the dashboard, IC2 and also on the hypervisor itself.

@TK_Liew - could you get someone looking into this, I can open a support ticket but it seems this should be simple enough to reproduce without needing access to a customer device given there are several of us seeing this.

1 Like

I’m about to set up fusion hub for a 310X, should I roll back to the prior firmware before doing so?

Standing by for an update before upgrading…

Running 40+ FusionHub Solo’s on several Hyper-V (2019) hosts, and upgraded all VM’s yesterday to 8.1.2. Not running into the same problem, CPU load is still 0-5% at idle per VM. Only 1 vCore per VM.

Hi all,

We do see the reported issue. Let us investigate then come back to you guys.

Thanks.

2 Likes

Hi all,

We found the problem and fixed it. We are preparing a special firmware. I will share it here once it is ready.

5 Likes

Thanks TK!

We spotted this because we have SNMP monitoring for all our hubs and threshold alarms as part of that.

As an aside would there be any scope to add CPU/Memory/Session Count logging and graphing to IC2 that would make spotting these sorts of things a lot easier for people who use only IC2 as their monitoring, can also help with spotting CPU/Memory related performance issues with busy hubs as it allows for correlation of those metrics vs traffic / peer count etc.

2 Likes

Hi all,

Here you go for the firmware to fix the issue of abnormal CPU usage when the FusionHub is idle.

https://download.peplink.com/firmware/fusionhub/fw-fusionhub_sfcn-8.1.2s030-build5006.bin

1 Like

CPU usage is supported by InControl2 for the time being.

1 Like

Thank for the quick turn around on the code. CPU usage back down to normal…

ScreenHunter_01 May. 24 05.07

2 Likes

We are seeing another issue since this firmware. According to inControl reports all the FusionHubs are using a lot more data than they are supposed to use.

We can see for example a Hub reporting a usage of 800GB in a month through the PepVPN, the device on the customer side is also reporting that usage on their WANs, so that seems to be right. But the reported usage of that FusionHub shows 5-6TB of usage for that month. Normally it would be double as you get download/upload. So how can this be? According to our DC the data has actually been used as well. So it is really strange how all these FusionHubs are using a lot more data than before…

It all started with 8.1.2, so I am flashing them all back to 8.1.1 for now.

cacti_graph_61

In the graph you can see the FusionHubs were using around 100Mbps constantly, and that went up to around 300Mbps constantly around the time the firmware was updated…

1 Like

You are on the fixed firmware? What is the exact version you are running?

8.1.2 build 5005 . We’d rather not run beta firmware on all of our units.

Did Peplink say Build 5006 is beta? I don’t think so. I’d update and see if it resolves.

I can’t select the firmware in inControl when pushing firmware, which normally means it’s a beta firmware. The option to enable beta firmware is gone from my inControl, so I can only push the firmware by entering a manual URL.

So no, I’d rather stick with a firmware on which we ran for months without issues and see if the issue is resolved.

Hello @padaco-daniel,
The “s” is for a “special” release, these are not published to InControl2 firmware list as they are often only done to address special circumstances, such as those of us with custom firmware for specific customer needs. We have, as do many other partners around the world, run with these “s” releases, these are safe to use and made available only on an as-need basis.

It is easy in InControl2 to use custom firmware.

  1. Choose “Custom” from the dropdown menu
  2. Enter the URL from above
    image
  3. Select Verify
    image
  4. Then choose when you want to commit the firmware and select Save Changes

If you want, you can treat the “s” as Safe instead of Special.
Happy to Help,
Marcus :slight_smile: