very basic outline shader

outline.fragment.glsl

#ifdef GL_ES
#define LOWP lowp
precision mediump float;
#else
#define LOWP
#endif

const float offset = 1.0 / 128.0;
varying vec2 v_texCoords;
uniform sampler2D u_texture;
void main()
{
	vec4 col = texture2D(u_texture, v_texCoords);
	if (col.a > 0.5)
		gl_FragColor = col;
	else {
		float a = texture2D(u_texture, vec2(v_texCoords.x + offset, v_texCoords.y)).a +
			texture2D(u_texture, vec2(v_texCoords.x, v_texCoords.y - offset)).a +
			texture2D(u_texture, vec2(v_texCoords.x - offset, v_texCoords.y)).a +
			texture2D(u_texture, vec2(v_texCoords.x, v_texCoords.y + offset)).a;
		if (col.a < 1.0 && a > 0.0)
			gl_FragColor = vec4(0.0, 0.0, 0.0, 0.8);
		else
			gl_FragColor = col;
	}
}

outline.vertex.glsl

attribute vec4 a_position;
attribute vec2 a_texCoord0;
uniform mat4 u_projTrans;
varying vec2 v_texCoords;

void main()
{
	v_texCoords = a_texCoord0;
	gl_Position =  u_projTrans * a_position;
}

I'm trying to rewrite this shader to Godot, and confused. How should I be converting this shader to Godot's GLSL Shader Language?

I'm trying to rewrite this shader to Godot, and confused. How should I be converting this shader to Godot's GLSL Shader Language?

1. The vertex-shader is standard behavior so you can omit it entirely.
2. In Godot the two

varying vec2 v_texCoords;
uniform sampler2D u_texture;

are built ins, so remove those and use their equivalents: v_texCoords -> UV, u_texture -> TEXTURE.
3. Instead of void main() in Godot the function void fragment() is used
4. texture instead of texture2D is used to sample a texture
5. Instead of hard-coding offset you can use the textureSize(...) function

A (more or less) direct port, where you can select the color and width of the outline via sliders:

shader_type canvas_item;

uniform float width : hint_range(0.0, 30.0);
uniform vec4 b_color : hint_color;

void fragment()
{
	float w = width * 1.0 / float(textureSize( TEXTURE, 0).x);
	float h = width * 1.0 / float(textureSize( TEXTURE, 0).y);
	vec4 color = texture(TEXTURE, UV);
	
	float alpha = -4.0 * color.a;
	alpha += texture(TEXTURE, UV + vec2( w, 0.0)).a;
	alpha += texture(TEXTURE, UV + vec2(-w, 0.0)).a;
	alpha += texture(TEXTURE, UV + vec2( 0.0, h)).a;
	alpha += texture(TEXTURE, UV + vec2( 0.0,-h)).a;
	vec4 final_color;
	if (color.a < 1.0 && alpha > 0.0) {
		final_color = b_color;
	} else {
		final_color = color;
	}

	COLOR = final_color;
}

Instead of the if, an interpolation gives better results if the edges of the sprite are anti-alized:

vec4 final_color = mix(color, b_color, clamp(alpha, 0.0, 1.0));
COLOR = vec4(final_color.rgb, clamp(abs(alpha) + color.a, 0.0, 1.0) );

One last hint: the outline shader require the image to have a transparent outline. If the sprite is cut to fit exactly the outline is out of bounds of the image and won't have any effect.

Well, I'm using this in Spatial shader type so will this work with SCREEN_TEXTURES?

I got to outline some 3d models, but the current create_outline_mesh just doesn't work in the way I wanted. Maybe I could look into it again, but will this shader work for 3d models?

Well, I'm using this in Spatial shader type so will this work with SCREEN_TEXTURES?

I never did anything 3D with godot but in theory, a shader like this could work, but would need to render to texture/canvas first so that you have the objects that you want an outline on isolated in that canvas, apply the shader to that canvas and then render that texture to the screen

Oh I'm bit confused now, I think someone could help me here.

Oh I'm bit confused now, I think someone could help me here.

This is a visiual illustration on how this shader works:

1. First we get which pixels in the texture are transparent (black)
2. We offset that to 4 directions: up (yellow), down (green), left (blue), right (red), in code that's the texture lookpus of the alpha channel e.g. texture(TEXTURE, UV + vec2( w, 0.0)).a
3. We combine those, that's why we sum up all the lookups
4. If the combination of those is above a certain threshold and the original texture is transparent on that position, we draw the outline color instead we draw the color from the original texture

In order to make the shader work in a 3D environment, you first need to render the objects that you want to have outlined into a texture, then render that texture onto the screen using the above shader.

Without detailed information about what you're trying to achieve and what your starting situation is, there is a lot of guesswork. So I suggest you ask your question in a forum where you can detail about the problem you're having.

I adapted it to gamemaker :D

varying vec2 v_vTexcoord;
varying vec4 v_vColour;

uniform float u_size;
uniform vec4 u_col; 

void main()
{
	vec4 col = texture2D(gm_BaseTexture, v_vTexcoord);
	if (col.a > 0.5)
	{
		gl_FragColor = col;
	}
	else 
	{
		float a = texture2D(gm_BaseTexture, vec2(v_vTexcoord.x + u_size, v_vTexcoord.y)).a +
			texture2D(gm_BaseTexture, vec2(v_vTexcoord.x, v_vTexcoord.y - u_size)).a +
			texture2D(gm_BaseTexture, vec2(v_vTexcoord.x - u_size, v_vTexcoord.y)).a +
			texture2D(gm_BaseTexture, vec2(v_vTexcoord.x, v_vTexcoord.y + u_size)).a;
		if (col.a < 1.0 && a > 0.0)
		{
			gl_FragColor = u_col;
		}
		else
		{
			gl_FragColor = col;
		}
	}
}